Skip to main content
 首页 » 编程设计

python之用于数据过滤的 TF-IDF

2025年01月19日10JeffreyZhao

我有一个原始文档列表,已经过滤并删除了英文停用词:

rawDocument = ['sport british english sports american english includes forms competitive physical activity games casual organised ...', 'disaster serious disruption occurring relatively short time functioning community society involving ...', 'government system group people governing organized community often state case broad associative definition ...', 'technology science craft greek τέχνη techne art skill cunning hand λογία logia collection techniques ...'] 

我用过

from sklearn.feature_extraction.text import TfidfVectorizer 
sklearn_tfidf = TfidfVectorizer(norm='l2', min_df=0, use_idf=True, smooth_idf=False, sublinear_tf=False) 
sklearn_representation = sklearn_tfidf.fit_transform(rawDocuments) 

但是我有一个

<4x50 sparse matrix of type '<class 'numpy.float64'>' 
    with 51 stored elements in Compressed Sparse Row format> 

我无法解释结果。那么,我是在使用正确的工具还是需要改变方式?

我的目标是获取每个文档中的相关词,以便与查询文档中的其他词执行余弦相似度。

提前谢谢你。

请您参考如下方法:

Pandas 模块通常可用于更好地可视化您的数据:

演示:

import pandas as pd 
 
df = pd.SparseDataFrame(sklearn_tfidf.fit_transform(rawDocument), 
                        columns=sklearn_tfidf.get_feature_names(), 
                        default_fill_value=0) 

结果:

In [85]: df 
Out[85]: 
   activity  american       art  associative  british    ...       system    techne  techniques  technology      time 
0      0.25      0.25  0.000000     0.000000     0.25    ...     0.000000  0.000000    0.000000    0.000000  0.000000 
1      0.00      0.00  0.000000     0.000000     0.00    ...     0.000000  0.000000    0.000000    0.000000  0.308556 
2      0.00      0.00  0.000000     0.282804     0.00    ...     0.282804  0.000000    0.000000    0.000000  0.000000 
3      0.00      0.00  0.288675     0.000000     0.00    ...     0.000000  0.288675    0.288675    0.288675  0.000000 
 
[4 rows x 48 columns]