Skip to main content
 首页 » 编程设计

python之分类标签文本的问题,错误的预测

2024年10月25日9sky-heaven

我正在使用 scikit-learn 提供的不同分类器和矢量化器,所以假设我有以下内容:

training = [["this was a good movie, 'POS'"], 
      ["this was a bad movie, 'NEG'"], 
      ["i went to the movies, 'NEU'"],  
      ["this movie was very exiting it was great, 'POS'"],  
      ["this is a boring film, 'NEG'"] 
        ,........................, 
          [" N-sentence, 'LABEL'"]] 
 
#Where each element of the list is another list that have documents, then. 
 
splitted = [#remove the tags from training] 
 
from sklearn.feature_extraction.text import HashingVectorizer 
X = HashingVectorizer( 
    tokenizer=lambda  doc: doc, lowercase=False).fit_transform(splitted) 
 
print X.toarray() 

然后我有这个矢量表示:

[[ 0.  0.  0. ...,  0.  0.  0.] 
 [ 0.  0.  0. ...,  0.  0.  0.] 
 [ 0.  0.  0. ...,  0.  0.  0.] 
 [ 0.  0.  0. ...,  0.  0.  0.] 
 [ 0.  0.  0. ...,  0.  0.  0.]] 

问题是我不知道我是否正确向量化了语料库,然后:

#This is the test corpus: 
test = ["I don't like this movie it sucks it doesn't liked me"] 
 
#I vectorize the corpus with hashing vectorizer 
Y = HashingVectorizer( 
    tokenizer=lambda  doc: doc, lowercase=False).fit_transform(test) 

然后我打印 Y :

[[ 0.  0.  0. ...,  0.  0.  0.]] 

然后

y = [x[-1]for x in training] 
 
#import SVM and classify 
from sklearn.svm import SVC 
svm = SVC() 
svm.fit(X, y) 
result = svm.predict(X) 
print "\nThe opinion is:\n",result 

这就是问题所在,我得到了 [NEG] 的以下 insted,这实际上是正确的预测:

["this was a good movie, 'POS'"] 

我想我没有正确矢量化 training或 y目标是错误的,任何人都可以帮助我了解发生了什么以及我应该如何向量化 training才能做出正确的预测?

请您参考如下方法:

我会留给你将训练数据转换成预期的格式:

training = ["this was a good movie", 
            "this was a bad movie", 
            "i went to the movies", 
            "this movie was very exiting it was great",  
            "this is a boring film"] 
 
labels = ['POS', 'NEG', 'NEU', 'POS', 'NEG'] 

特征提取

>>> from sklearn.feature_extraction.text import HashingVectorizer 
>>> vect = HashingVectorizer(n_features=5, stop_words='english', non_negative=True) 
>>> X_train = vect.fit_transform(training) 
>>> X_train.toarray() 
[[ 0.          0.70710678  0.          0.          0.70710678] 
 [ 0.70710678  0.70710678  0.          0.          0.        ] 
 [ 0.          0.          0.          0.          0.        ] 
 [ 0.          0.89442719  0.          0.4472136   0.        ] 
 [ 1.          0.          0.          0.          0.        ]] 

对于更大的语料库,您应该增加 n_features 以避免冲突,我使用 5 以便可以可视化生成的矩阵。另请注意,我使用了 stop_words='english',我认为在如此少的示例中,删除停用词很重要,否则您可能会混淆分类器。

模型训练

from sklearn.svm import SVC 
 
model = SVC() 
model.fit(X_train, labels) 

预测

>>> test = ["I don't like this movie it sucks it doesn't liked me"] 
>>> X_pred = vect.transform(test) 
>>> model.predict(X_pred) 
['NEG'] 
 
>>> test = ["I think it was a good movie"] 
>>> X_pred = vect.transform(test) 
>>> model.predict(X_pred) 
['POS'] 

编辑:请注意,第一个测试示例的正确分类只是一个幸运的巧合,因为我没有看到任何可以从训练集中学到的单词是否定的。在第二个示例中,单词 good 可能会触发正面分类。