Skip to main content
 首页 » 编程设计

python之两个列表元素的PMI

2025年04月02日9shangdawei

我想计算两个列表元素的逐点互信息分数。 假设我们有

ListA = "Hi there, This is only a test message. Please enjoy the weather in the park." 
ListB = "work, bank, tree, weather, sun" 

然后我如何计算所有对 (work, Hi), (work, there), (work, This) .... (sun, park) 的 PMI 分数。

它对我计算一个列表的二元组的 PMI 很有用:

def pmi(word1, word2, unigram_freq, bigram_freq, unigram_freq_values, bigram_freq_values, output_name): 
    prob_word1 = unigram_freq[word1] / float(sum(unigram_freq_values)) 
    prob_word2 = unigram_freq[word2] / float(sum(unigram_freq_values)) 
    prob_word1_word2 = bigram_freq / float(sum(bigram_freq_values)) 
    pmi =  math.log(prob_word1_word2/float(prob_word1*prob_word2),2) 
 
unigrams = nltk.FreqDist(ListA) 
bigrams = ngrams(ListA,2) 
 
n1_freq = nltk.FreqDist(unigrams) 
n2_freq = nltk.FreqDist(bigrams) 
 
output_pmi = "test.txt" 
for bigram, freq in n2_freq.most_common(1000): 
    w1 = bigram[0] 
    w2 = bigram[1] 
    unigram_freq_val = n1_freq.values() 
    bigram_freq_val = n2_freq.values() 
    pmi(w1, w2, unigrams, freq, unigram_freq_val, bigram_freq_val, output_pmi)  

我在计算 ListA 和 ListB 的二元组的 PMI 时遇到了问题。如果有人能帮助我,我将不胜感激。非常感谢!

(当然,这两个列表只是我的任务的最小示例。)

请您参考如下方法:

如果您要查找两个列表的所有组合:

import itertools 
 
ListA = "Hi there, This is only a test message. Please enjoy the weather in the park." 
ListB = "work, bank, tree, weather, sun" 
WordsA = ListA.split() 
WordsB = ListB.split() 
#print(WordsA, "\n\n", WordsB)              #This is to show what the new lists are 
c = list(itertools.product(WordsA, WordsB)) 
print(c)