Skip to main content
 首页 » 编程设计

python之"Phrase"停用词在 sklearn/nltk 中被忽略

2025年01月19日36yxwkf

我有一个流程是这样的:

  • 构建单词标记模式 - 天真地,2 个以上的字母数字被单词边界包围。
  • 标记文档,然后使用 nltk 对这些标记进行词形还原。
  • 为 sklearn 的内置英语停用词添加一些“自定义”停用词。 (此处,在可重现的示例中仅使用一个公司名称。)
  • 利用上述方法获取词频,从 unigrams 到 4-grams。

问题是(大概是因为标记化先发生的事实?)多词停用词(短语)没有被删除。

完整示例:

import re 
import nltk 
from nltk.stem import WordNetLemmatizer 
from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS as ESW, CountVectorizer 
 
# Make sure we have the corpora used by nltk's lemmatizer 
try: 
    nltk.data.find('corpora/wordnet') 
except: 
    nltk.download('wordnet') 
 
# "Naive" token similar to that used by sklearn 
TOKEN = re.compile(r'\b\w{2,}\b') 
 
# Tokenize, then lemmatize these tokens 
# Modified from: 
# http://scikit-learn.org/stable/modules/feature_extraction.html#customizing-the-vectorizer-classes 
class LemmaTokenizer(object): 
    def __init__(self): 
        self.wnl = WordNetLemmatizer() 
    def __call__(self, doc): 
        return (self.wnl.lemmatize(t) for t in TOKEN.findall(doc)) 
 
# Add 1 more phrase to sklearn's stop word list 
sw = ESW.union(frozenset(['sinclair broadcast group'])) 
 
vect = CountVectorizer(stop_words=sw, ngram_range=(1, 4), 
                       tokenizer=LemmaTokenizer()) 
 
# These are nonsense babbling 
docs = ["""And you ask Why You Are Sinclair Broadcast Group is Asking It""", 
        """Why are you asking what Sinclair Broadcast Group and you"""] 
 
tf = vect.fit_transform(docs) 

重申一下:单词停用词已被正确删除,但短语仍然存在:

vect.get_feature_names() 
 
# ['ask', 
#  'ask sinclair', 
#  'ask sinclair broadcast', 
#  'ask sinclair broadcast group', 
#  'asking', 
#  'asking sinclair', 
#  'asking sinclair broadcast', 
#  'asking sinclair broadcast group', 
#  'broadcast', 
#  'broadcast group', 
#  'broadcast group asking', 
#  'group', 
#  'group asking', 
#  'sinclair', 
#  'sinclair broadcast', 
#  'sinclair broadcast group', 
#  'sinclair broadcast group asking'] 

我该如何纠正这个问题?

请您参考如下方法:

来自 the documentation of CountVectorizer :

stop_words : string {‘english’}, list, or None (default)

If ‘english’, a built-in stop word list for English is used.

If a list, that list is assumed to contain stop words, all of which will be removed from the resulting tokens. Only applies if analyzer == 'word'.

If None, no stop words will be used. max_df can be set to a value in the range [0.7, 1.0) to automatically detect and filter stop words based on intra corpus document frequency of terms.

再往下看参数token_pattern:

token_pattern : string

Regular expression denoting what constitutes a “token”, only used if analyzer == 'word'. The default regexp select tokens of 2 or more alphanumeric characters (punctuation is completely ignored and always treated as a token separator).

因此,如果 analyzer(token) 的结果等于 'sinclair broadcast group',它只会删除停用词。但默认的 analyzer 是 'word',这意味着停用词检测仅适用于单个单词,因为标记由默认的 token_pattern 定义如上所述。

标记不是 n-gram(相反,n-gram 是由标记组成的,停用词删除似乎发生在标记级别,先于构建 n-gram)。

作为快速检查,您可以将自定义停用词更改为 'sinclair' 以进行实验,以查看它是否可以在将该词视为孤立词时正确删除该词。

换句话说,您需要将自己的可调用对象作为 analyzer 传递,以使其也将分析器逻辑应用于 n-gram,您必须手动检查它。但默认行为假定停用词检测不能应用于 n-gram,只能应用于单个词。

以下是针对您的案例的自定义分析器函数示例。这是 based on this answer ...请注意,我没有对其进行测试,因此可能存在错误。

def trigram_match(i, trigram, words): 
    if i < len(words) - 2 and words[i:i + 3] == trigram: 
        return True 
    if (i > 0 and i < len(words) - 1) and words[i - 1:i + 2] == trigram: 
        return True 
    if i > 1 and words[i - 2:i + 1] == trigram: 
        return True 
    return False 
     
 
def custom_analyzer(text): 
    bad_trigram = ['sinclair', 'broadcasting', 'group'] 
    words = [str.lower(w) for w in re.findall(r'\w{2,}', text)] 
    for i, w in enumerate(words): 
        if w in sw or trigram_match(i, bad_trigram, words): 
            continue 
        yield w