我有一个流程是这样的:
- 构建单词标记模式 - 天真地,2 个以上的字母数字被单词边界包围。
- 标记文档,然后使用 nltk 对这些标记进行词形还原。
- 为 sklearn 的内置英语停用词添加一些“自定义”停用词。 (此处,在可重现的示例中仅使用一个公司名称。)
- 利用上述方法获取词频,从 unigrams 到 4-grams。
问题是(大概是因为标记化先发生的事实?)多词停用词(短语)没有被删除。
完整示例:
import re
import nltk
from nltk.stem import WordNetLemmatizer
from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS as ESW, CountVectorizer
# Make sure we have the corpora used by nltk's lemmatizer
try:
nltk.data.find('corpora/wordnet')
except:
nltk.download('wordnet')
# "Naive" token similar to that used by sklearn
TOKEN = re.compile(r'\b\w{2,}\b')
# Tokenize, then lemmatize these tokens
# Modified from:
# http://scikit-learn.org/stable/modules/feature_extraction.html#customizing-the-vectorizer-classes
class LemmaTokenizer(object):
def __init__(self):
self.wnl = WordNetLemmatizer()
def __call__(self, doc):
return (self.wnl.lemmatize(t) for t in TOKEN.findall(doc))
# Add 1 more phrase to sklearn's stop word list
sw = ESW.union(frozenset(['sinclair broadcast group']))
vect = CountVectorizer(stop_words=sw, ngram_range=(1, 4),
tokenizer=LemmaTokenizer())
# These are nonsense babbling
docs = ["""And you ask Why You Are Sinclair Broadcast Group is Asking It""",
"""Why are you asking what Sinclair Broadcast Group and you"""]
tf = vect.fit_transform(docs)
重申一下:单词停用词已被正确删除,但短语仍然存在:
vect.get_feature_names()
# ['ask',
# 'ask sinclair',
# 'ask sinclair broadcast',
# 'ask sinclair broadcast group',
# 'asking',
# 'asking sinclair',
# 'asking sinclair broadcast',
# 'asking sinclair broadcast group',
# 'broadcast',
# 'broadcast group',
# 'broadcast group asking',
# 'group',
# 'group asking',
# 'sinclair',
# 'sinclair broadcast',
# 'sinclair broadcast group',
# 'sinclair broadcast group asking']
我该如何纠正这个问题?
请您参考如下方法:
来自 the documentation of CountVectorizer :
stop_words : string {‘english’}, list, or None (default)
If ‘english’, a built-in stop word list for English is used.
If a list, that list is assumed to contain stop words, all of which will be removed from the resulting tokens. Only applies if analyzer == 'word'.
If None, no stop words will be used. max_df can be set to a value in the range [0.7, 1.0) to automatically detect and filter stop words based on intra corpus document frequency of terms.
再往下看参数token_pattern:
token_pattern : string
Regular expression denoting what constitutes a “token”, only used if analyzer == 'word'. The default regexp select tokens of 2 or more alphanumeric characters (punctuation is completely ignored and always treated as a token separator).
因此,如果 analyzer(token) 的结果等于 'sinclair broadcast group',它只会删除停用词。但默认的 analyzer 是 'word',这意味着停用词检测仅适用于单个单词,因为标记由默认的 token_pattern 定义如上所述。
标记不是 n-gram(相反,n-gram 是由标记组成的,停用词删除似乎发生在标记级别,先于构建 n-gram)。
作为快速检查,您可以将自定义停用词更改为 'sinclair' 以进行实验,以查看它是否可以在将该词视为孤立词时正确删除该词。
换句话说,您需要将自己的可调用对象作为 analyzer 传递,以使其也将分析器逻辑应用于 n-gram,您必须手动检查它。但默认行为假定停用词检测不能应用于 n-gram,只能应用于单个词。
以下是针对您的案例的自定义分析器函数示例。这是 based on this answer ...请注意,我没有对其进行测试,因此可能存在错误。
def trigram_match(i, trigram, words):
if i < len(words) - 2 and words[i:i + 3] == trigram:
return True
if (i > 0 and i < len(words) - 1) and words[i - 1:i + 2] == trigram:
return True
if i > 1 and words[i - 2:i + 1] == trigram:
return True
return False
def custom_analyzer(text):
bad_trigram = ['sinclair', 'broadcasting', 'group']
words = [str.lower(w) for w in re.findall(r'\w{2,}', text)]
for i, w in enumerate(words):
if w in sw or trigram_match(i, bad_trigram, words):
continue
yield w
