python之Tfidfvectorizer之如何查看已处理的 token

如何检查 TfidfVertorizer() 中标记化的字符串？如果我没有在参数中传递任何内容，TfidfVertorizer() 将使用一些预定义的方法标记字符串。我想观察它如何标记字符串，以便我可以更轻松地调整我的模型。

from sklearn.feature_extraction.text import TfidfVectorizer 
corpus = ['This is the first document.', 
          'This document is the second document.', 
          'And this is the third one.', 
          'Is this the first document?'] 
vectorizer = TfidfVectorizer() 
X = vectorizer.fit_transform(corpus) 

我想要这样的东西:

>>>vectorizer.get_processed_tokens() 
[['this', 'is', 'first', 'document'], 
 ['this', 'document', 'is', 'second', 'document'], 
 ['this', 'is', 'the', 'third', 'one'], 
 ['is', 'this', 'the', 'first', 'document']] 

我该怎么做？

请您参考如下方法:

build_tokenizer()正好可以达到这个目的。

试试这个!

tokenizer = lambda docs: [vectorizer.build_tokenizer()(doc) for doc in docs] 
 
tokenizer(corpus) 

输出:

[['This', 'is', 'the', 'first', 'document'], 
 ['This', 'document', 'is', 'the', 'second', 'document'], 
 ['And', 'this', 'is', 'the', 'third', 'one'], 
 ['Is', 'this', 'the', 'first', 'document']] 

一个类轮解决方案是

list(map(vectorizer.build_tokenizer(),corpus))