我想计算两个列表元素的逐点互信息分数。 假设我们有
ListA = "Hi there, This is only a test message. Please enjoy the weather in the park."
ListB = "work, bank, tree, weather, sun"
然后我如何计算所有对 (work, Hi), (work, there), (work, This) .... (sun, park) 的 PMI 分数。
它对我计算一个列表的二元组的 PMI 很有用:
def pmi(word1, word2, unigram_freq, bigram_freq, unigram_freq_values, bigram_freq_values, output_name):
prob_word1 = unigram_freq[word1] / float(sum(unigram_freq_values))
prob_word2 = unigram_freq[word2] / float(sum(unigram_freq_values))
prob_word1_word2 = bigram_freq / float(sum(bigram_freq_values))
pmi = math.log(prob_word1_word2/float(prob_word1*prob_word2),2)
unigrams = nltk.FreqDist(ListA)
bigrams = ngrams(ListA,2)
n1_freq = nltk.FreqDist(unigrams)
n2_freq = nltk.FreqDist(bigrams)
output_pmi = "test.txt"
for bigram, freq in n2_freq.most_common(1000):
w1 = bigram[0]
w2 = bigram[1]
unigram_freq_val = n1_freq.values()
bigram_freq_val = n2_freq.values()
pmi(w1, w2, unigrams, freq, unigram_freq_val, bigram_freq_val, output_pmi)
我在计算 ListA 和 ListB 的二元组的 PMI 时遇到了问题。如果有人能帮助我,我将不胜感激。非常感谢!
(当然,这两个列表只是我的任务的最小示例。)
请您参考如下方法:
如果您要查找两个列表的所有组合:
import itertools
ListA = "Hi there, This is only a test message. Please enjoy the weather in the park."
ListB = "work, bank, tree, weather, sun"
WordsA = ListA.split()
WordsB = ListB.split()
#print(WordsA, "\n\n", WordsB) #This is to show what the new lists are
c = list(itertools.product(WordsA, WordsB))
print(c)
