Skip to main content
 首页 » 编程设计

具有预定义单词的 Python 单词频率

2025年02月15日14落叶

我在一个文本文件中有一组数据,我想根据预定义的词(drive、street、i、lives)构建一个频率表。下面是例子

 ID |  Text 
 ---|-------------------------------------------------------------------- 
 1  | i drive to work everyday in the morning and i drive back in the evening on main street 
 2  | i drive back in a car and then drive to the gym on 5th street 
 3  | Joe lives in Newyork on NY street 
 4  | Tod lives in Jersey city on NJ street 

这里是我想得到的输出

ID  |  drive |  street  |   i  |  lives 
----|--------|----------|------|------- 
1   |   2    |    1     |   2  |   0 
2   |   2    |    1     |   1  |   0 
3   |   0    |    1     |   0  |   1 
4   |   0    |    1     |   0  |   1 

这是我正在使用的代码,我可以找到单词的数量,但这并不能解决我的需求,我想使用一组预定义的单词来找到如上所示的计数

   from nltk.corpus import stopwords 
   import string 
   from collections import Counter 
   import nltk 
   from nltk.tag import pos_tag 
 
   xy = open('C:\Python\data\file.txt').read().split() 
   q = (w.lower() for w in xy) 
 
   stopset = set(stopwords.words('english')) 
 
   filtered_words = [word for word in xyz if not word  in stopset] 
   filtered_words = [] 
   for word in xyz: 
       if word not in stopset: 
       filtered_words.append(word) 
   print(Counter(filtered_words)) 
   print(len(filtered_words)) 

请您参考如下方法:

sklearn.feature_extraction.text.CountVectorizer 之类的东西似乎与您正在寻找的东西很接近。此外,collections.Counter 可能会有帮助。你打算如何使用这个数据结构?如果您碰巧尝试进行机器学习/预测,那么值得研究一下 sklearn.feature_extraction.text 中的不同矢量化器。

编辑:

text = ['i drive to work everyday in the morning and i drive back in the evening on main street', 
        'i drive back in a car and then drive to the gym on 5th street', 
        'Joe lives in Newyork on NY street', 
        'Tod lives in Jersey city on NJ street'] 
 
from sklearn.feature_extraction.text import CountVectorizer 
 
vocab = ['drive', 'street', 'i', 'lives'] 
 
vectorizer = CountVectorizer(vocabulary = vocab) 
 
# turn the text above into a matrix of shape R X C 
# where R is number of rows (elements in your text array) 
# and C is the number of elements in the set of all words in your text array 
X = vectorizer.fit_transform(text) 
 
# sparse to dense matrix 
X = X.toarray() 
 
# get the feature names from the already-fitted vectorizer 
vectorizer_feature_names = vectorizer.get_feature_names() 
 
# prove that the vectorizer's feature names are identical to the vocab you specified above 
assert vectorizer_feature_names == vocab 
 
# make a table with word frequencies as values and vocab as columns 
out_df = pd.DataFrame(data = X, columns = vectorizer_feature_names) 
 
print(out_df) 

然后,你的结果:

       drive  street  i  lives 
    0      2       1  0      0 
    1      2       1  0      0 
    2      0       1  0      1 
    3      0       1  0      1