Skip to main content
 首页 » 编程设计

python之NLTK 语料库路透社中类别的含义是什么

2025年05月04日9java哥

我在做文本主题分类时遇到了问题。

我得到了 NLTK“路透社”语料库中的数据..

但是当我尝试“reuters.categories()”时

结果是

['acq', 'alum', 'barley', 'bop', 'carcass', 'castor-oil', 'cocoa', 'coconut', 'coconut-oil', 'coffee', 'copper ', 'copra-cake', 'corn', 'cotton', 'cotton-oil', 'cpi', 'cpu', 'crude', 'dfl', 'dlr', 'dmk', 'earn', '燃料'、'天然气'、'gnp'、'黄金'、' Cereal '、'花生'、'花生油'、'热'、'生 pig '、'住房'、'收入'、'安装债务', 'interest', 'ipi', 'iron-steel', 'jet', 'jobs', 'l-cattle', 'lead', 'lei', 'lin-oil', 'livestock', 'lumber ','膳食饲料','货币外汇','货币供应','石脑油','天然气','镍','nkr','nzdlr','燕麦','油籽', 'orange', 'palladium', 'palm-oil', 'palmkernel', 'pet-chem', 'platinum', 'potato', 'propane', 'rand', 'rape-oil', 'rapeseed', '储备', '零售', '大米', '橡胶', '黑麦', '船', '银', '高粱', ' bean 粕', ' bean 油', '大 bean ', '战略-metal', 'sugar', 'sun-meal', 'sun-oil', 'sunseed', 'tea', 'tin', 'trade', 'veg-oil', 'wheat', 'wpi', '日元', '锌']

我几乎不知道每一个是什么意思,我能找到一些解释吗?

请您参考如下方法:

NLTK语料库API中路透社语料库的相关信息:

  • Reuters-21578“ApteMod”语料库专为文本分类而构建。

  • ApteMod 是来自路透社的 10,788 份文件的集合 财经新闻专线服务

  • 在 ApteMod 语料库中,每个文档属于一个或多个类别。语料库中有 90 个类别。

文件 ID 到类别的映射可以在 ~/nltk_data/corpora/reuters/cats.txt

中找到
from os.path import expanduser 
from collections import defaultdict 
from nltk.corpus import reuters 
 
home = expanduser("~") 
id2cat = defaultdict(list) 
 
for line in open(home+'/nltk_data/corpora/reuters/cats.txt','r'): 
    fid, _, cats = line.partition(' ') 
    id2cat[fid] = cats.split() 
 
for fileid in reuters.fileids(): 
    for sent in reuters.sents(fileid): 
        print id2cat[fileid], sent 

[输出]:

['trade'] ['ASIAN', 'EXPORTERS', 'FEAR', 'DAMAGE', 'FROM', 'U', '.', 'S', '.-', 'JAPAN', 'RIFT', 'Mounting', 'trade', 'friction', 'between', 'the', 'U', '.', 'S', '.', 'And', 'Japan', 'has', 'raised', 'fears', 'among', 'many', 'of', 'Asia', "'", 's', 'exporting', 'nations', 'that', 'the', 'row', 'could', 'inflict', 'far', '-', 'reaching', 'economic', 'damage', ',', 'businessmen', 'and', 'officials', 'said', '.'] 
... 

你可以从这个文件中找到关于类别的信息:~/nltk_data/corpora/reuters/README:

  The Reuters-21578 benchmark corpus, ApteMod version 

This is a publically available version of the well-known Reuters-21578 "ApteMod" corpus for text categorization. It has been used in publications like these:

ApteMod is a collection of 10,788 documents from the Reuters financial newswire service, partitioned into a training set with 7769 documents and a test set with 3019 documents. The total size of the corpus is about 43 MB. It is also available for download from http://kdd.ics.uci.edu/databases/reuters21578/reuters21578.html , which includes a more extensive history of the data revisions.

The distribution of categories in the ApteMod corpus is highly skewed, with 36.7% of the documents in the most common category, and only 0.0185% (2 documents) in each of the five least common categories. In fact, the original data source is even more skewed---in creating the corpus, any categories that did not contain at least one document in the training set and one document in the test set were removed from the corpus by its original creator.

In the ApteMod corpus, each document belongs to one or more categories. There are 90 categories in the corpus. The average number of categories per document is 1.235, and the average number of documents per category is about 148, or 1.37% of the corpus.

-Ken Williams ken@mathforum.org

     Copyright & Notification  

(extracted from the README at the UCI address above)

The copyright for the text of newswire articles and Reuters annotations in the Reuters-21578 collection resides with Reuters Ltd. Reuters Ltd. and Carnegie Group, Inc. have agreed to allow the free distribution of this data for research purposes only.

If you publish results based on this data set, please acknowledge its use, refer to the data set by the name "Reuters-21578, Distribution 1.0", and inform your readers of the current location of the data set (see "Availability & Questions").