python之使用 pyspark 计算所有可能的单词对_编程设计

我有一个文本文档。我需要找到整个文档中重复单词对的可能计数。例如，我有以下 word 文档。该文档有两行，每行以“;”分隔。文档:

My name is Sam My name is Sam My name is Sam; 
My name is Sam;

我正在研究成对词计数。预期结果是:

[(('my', 'my'), 3), (('name', 'is'), 7), (('is', 'name'), 3), (('sam', 'sam'), 3), (('my', 'name'), 7), (('name', 'sam'), 7), (('is', 'my'), 3), (('sam', 'is'), 3), (('my', 'sam'), 7), (('name', 'name'), 3), (('is', 'is'), 3), (('sam', 'my'), 3), (('my', 'is'), 7), (('name', 'my'), 3), (('is', 'sam'), 7), (('sam', 'name'), 3)]

如果我使用:

wordPairCount = rddData.map(lambda line: line.split()).flatMap(lambda x: [((x[i], x[i + 1]), 1) for i in range(0, len(x) - 1)]).reduceByKey(lambda a,b:a + b)

我得到连续单词的成对单词及其重复出现次数。

如何将行中的每个词与其他词配对，然后在所有行中搜索同一对？

有人可以看看吗？谢谢

请您参考如下方法:

您输入的字符串:

# spark is SparkSession object 
s1 = 'The Adventure of the Blue Carbuncle The Adventure of the Blue Carbuncle The Adventure of the Blue Carbuncle; The Adventure of the Blue Carbuncle;' 
 
# Split the string on ; and I parallelize it to make an rdd 
rddData = spark.sparkContext.parallelize(rdd_Data.split(";")) 
 
rddData.collect() 
# ['The Adventure of the Blue Carbuncle The Adventure of the Blue Carbuncle The Adventure of the Blue Carbuncle', ' The Adventure of the Blue Carbuncle', ''] 
 
import itertools 
 
final = ( 
    rddData.filter(lambda x: x != "") 
        .map(lambda x: x.split(" ")) 
        .flatMap(lambda x: itertools.combinations(x, 2)) 
        .filter(lambda x: x[0] != "") 
        .map(lambda x: (x, 1)) 
        .reduceByKey(lambda x, y: x + y).collect() 
) 
# [(('The', 'of'), 7), (('The', 'Blue'), 7), (('The', 'Carbuncle'), 7), (('Adventure', 'the'), 7), (('Adventure', 'Adventure'), 3), (('of', 'The'), 3), (('the', 'Adventure'), 3), (('the', 'the'), 3), (('Blue', 'The'), 3), (('Carbuncle', 'The'), 3), (('Adventure', 'The'), 3), (('of', 'the'), 7), (('of', 'Adventure'), 3), (('the', 'The'), 3), (('Blue', 'Adventure'), 3), (('Blue', 'the'), 3), (('Carbuncle', 'Adventure'), 3), (('Carbuncle', 'the'), 3), (('The', 'The'), 3), (('of', 'Blue'), 7), (('of', 'Carbuncle'), 7), (('of', 'of'), 3), (('Blue', 'Carbuncle'), 7), (('Blue', 'of'), 3), (('Blue', 'Blue'), 3), (('Carbuncle', 'of'), 3), (('Carbuncle', 'Blue'), 3), (('Carbuncle', 'Carbuncle'), 3), (('The', 'Adventure'), 7), (('The', 'the'), 7), (('Adventure', 'of'), 7), (('Adventure', 'Blue'), 7), (('Adventure', 'Carbuncle'), 7), (('the', 'Blue'), 7), (('the', 'Carbuncle'), 7), (('the', 'of'), 3)]

从第一个拆分中删除所有空格
split x 这是一个空格分隔的字符串，按空格
使用 itertools.combinations 创建 2 个元素的组合(flatMap 用于将每个单词与行中的每个其他单词配对)
像计算字数一样进行映射和缩减