为了对文本进行分类,我想使用SVM。我想将测试数据分类为标签之一(健康/成人)培训和测试数据是文本文件
我正在使用python的scikit库。当我将文本保存到txt文件时,我对其进行utf-8
了编码,这就是为什么我要在代码段中对其进行解码。这是我尝试的代码
String = String.decode('utf-8')
String2 = String2.decode('utf-8')
bigram_vectorizer = CountVectorizer(ngram_range=(1, 2),
token_pattern=r'\b\w+\b', min_df=1)
X_2 = bigram_vectorizer.fit_transform(String2).toarray()
X_1 = bigram_vectorizer.fit_transform(String).toarray()
X_train = np.array([X_1,X_2])
print type(X_train)
y = np.array([1, 2])
clf = SVC()
clf.fit(X_train, y)
#prepare test data
print(clf.predict(X))
这是我得到的错误
File "/Users/guru/python_projects/implement_LDA/lda/apply.py", line 107, in <module>
clf.fit(X_train, y)
File "/Users/guru/python_projects/implement_LDA/lda/lib/python2.7/site-packages/sklearn/svm/base.py", line 150, in fit
X = check_array(X, accept_sparse='csr', dtype=np.float64, order='C')
File "/Users/guru/python_projects/implement_LDA/lda/lib/python2.7/site-packages/sklearn/utils/validation.py", line 373, in check_array
array = np.array(array, dtype=dtype, order=order, copy=copy)
ValueError: setting an array element with a sequence.
当我搜索错误时,我发现了一些结果,但它们甚至都没有帮助。在应用SVM模型时,我认为这在逻辑上是错误的。有人可以给我一个提示吗?
您必须合并样本,将其向量化,然后拟合分类器。像这样:
String = String.decode('utf-8')
String2 = String2.decode('utf-8')
bigram_vectorizer = CountVectorizer(ngram_range=(1, 2),
token_pattern=r'\b\w+\b', min_df=1)
X_train = bigram_vectorizer.fit_transform(np.array([String, String2]))
print type(X_train)
y = np.array([1, 2])
clf = SVC()
clf.fit(X_train, y)
#prepare test data
print(clf.predict(bigram_vectorizer.transform(np.array([X1, X2, ...]))))
但是2个样本的数据量很少,因此您的预测可能不准确。
编辑:
您也可以使用管道在一个步骤中组合转换和分类。
from sklearn.pipeline import Pipeline
print type(X_train) # Should be a list of texts length 100 in your case
y_train = ... # Should be also a list of length 100
clf = Pipeline([
('transformer', CountVectorizer(...)),
('estimator', SVC()),
])
clf.fit(X_train, y_train)
X_test = np.array(["sometext"]) # array of test texts length = 1
print(clf.predict(X_test))
本文收集自互联网,转载请注明来源。
如有侵权,请联系[email protected] 删除。
我来说两句