keras 多文本特征输入和单文本标签输出分类

数据挖掘 喀拉斯 张量流
2022-02-14 03:59:26

我正在努力寻找在特征输入和单个输出文本标签的数据框中使用多个文本列预处理文本数据的确切方法。下面是示例数据集,

name    |   big_text_phrase     |   action      | action_type
--------------------------------------------------------------
test_name_1 | some bag of words |   action_1    | action_type_1
test_name_2 | some different bag of words   |   action_2    | action_type_2

当我尝试仅使用一个特征 big_text_phrase 作为输入和输出标签作为名称进行文本分类时,它工作正常并且能够预测。下面是具有单个文本特征输入的模型详细信息。

import numpy as np
import pandas as pd
import tensorflow as tf

from sklearn.preprocessing import LabelBinarizer, LabelEncoder
from sklearn.metrics import confusion_matrix

from tensorflow import keras
from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout
from keras.preprocessing import text, sequence
from keras import utils
train_size = int(len(data) * .8)
train_posts = data['big_text_phrase'][:train_size]
train_tags = data['name'][:train_size]
test_posts = data['big_text_phrase'][train_size:]
test_tags = data['name'][train_size:]
max_words = 10000
tokenize = text.Tokenizer(num_words=max_words, char_level=False)
tokenize.fit_on_texts(train_posts)
x_train = tokenize.texts_to_matrix(train_posts)
x_test = tokenize.texts_to_matrix(test_posts)
encoder = LabelEncoder()
encoder.fit(train_tags)
y_train = encoder.transform(train_tags)
y_test = encoder.transform(test_tags)
num_classes = np.max(y_train) + 1
y_train = utils.to_categorical(y_train, num_classes)
y_test = utils.to_categorical(y_test, num_classes)
batch_size = 32
model = Sequential()
model.add(Dense(512, input_shape=(max_words,)))
model.add(Activation('sigmoid'))
model.add(Dropout(0.5))
model.add(Dense(num_classes))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
history = model.fit(x_train, y_train, batch_size=batch_size, epochs=5, verbose=1, validation_split=0.1)
score = model.evaluate(x_test, y_test, batch_size=batch_size, verbose=1)

但是对于如何使用多个输入文本功能和单个输出文本标签来实现相同的功能充满困惑,谷歌搜索并没有得到太多细节,并且听说一种热编码可能适合。所以我尝试了一种热门的分类方法。由于我在 big_text_phrase 列中有大量单词,并且分类列正在增长。

更新1:

我尝试了一些不同的方法(可能是愚蠢的尝试),如下所示样本计数出错的地方,

train_posts = data[['big_text_phrase', 'action', 'action_type']][:train_size]
train_tags = data['name'][:train_size]
test_posts = data[['big_text_phrase', 'action', 'action_type']][train_size:]
test_tags = data['name'][train_size:]

print('x_train shape:', x_train.shape)
print('x_test shape:', x_test.shape)
print('y_train shape:', y_train.shape)
print('y_test shape:', y_test.shape)
x_train shape: (3, 10000)
x_test shape: (3, 10000)
y_train shape: (3587, 47)
y_test shape: (897, 47)

模型返回错误Input arrays should have the same number of samples as target arrays. Found 3 input samples and 3587 target samples.

0个回答
没有发现任何回复~