Skip to main content
 首页 » 编程设计

python之如何在 python 中计算单词的上限

2024年11月01日9insus

我正在写一个代码,来计算一个包含大约 20,000 个文件的文档中单词出现的频率,我能够得到一个单词在文档中的整体频率并且 到目前为止我的代码是:

import os 
import re 
import sys 
sys.stdout=open('f2.txt','w') 
from collections import Counter 
from glob import iglob 
 
def removegarbage(text): 
    text=re.sub(r'\W+',' ',text) 
    text=text.lower() 
    return text 
 
folderpath='d:/articles-words' 
counter=Counter() 
d=0 
 
for filepath in iglob(os.path.join(folderpath,'*.txt')): 
    with open(filepath,'r') as filehandle: 
        d+=1 
 
r=round(d*0.1) 
for filepath in iglob(os.path.join(folderpath,'*.txt')): 
    with open(filepath,'r') as filehandle: 
        words=set(removegarbage(filehandle.read()).split()) 
        if r > counter: 
             counter.update() 
 
for word,count in counter.most_common(): 
    print('{}  {}'.format(word,count)) 

但是,我想修改我的计数器,只有当计数大于 r=0.1*(文件数)时才更新它 简而言之,我想阅读在整个文档中频率大于文档数量10%的单词。 错误是:

Traceback (most recent call last): 
  File "<stdin>", line 4, in <module> 
TypeError: unorderable types: int() > Counter() 

如何修改?

请您参考如下方法:

这样的事情怎么样?

from glob import glob # (instead of iglob) 
 
... 
 
filepaths = glob(os.path.join(folderpath,'*.txt')) 
 
num_files = len(filepaths) 
 
# Add all words to counter 
for filepath in filepaths): 
    with open(filepath,'r') as filehandle: 
        lines = filehandle.read() 
        words = removegarbage(lines).split() 
        counter.update(words) 
 
# Display most common 
for word, count in counter.most_common(): 
 
    # Break out if the frequency is less than 0.1 * the number of files 
    if count < 0.1*num_files: 
        break 
 
    print('{}  {}'.format(word,count)) 

Counter.iteritems()的使用:

>>> from collections import Counter 
>>> c = Counter() 
>>> c.update(['test', 'test', 'test2']) 
>>> c.iteritems() 
<dictionary-itemiterator object at 0x012F4750> 
>>> for word, count in c.iteritems(): 
...     print word, count 
...      
test 2 
test2 1