python之在 PySpark 中将值添加到 DenseVector

我有一个经过处理的 DataFrame:

+---------+-------+ 
| inputs  | temp  |  
+---------+-------+ 
| [1,0,0] | 12    | 
+---------+-------+ 
| [0,1,0] | 10    | 
+---------+-------+ 
... 

inputs 是 DenseVectors 的一列。 temp 是一列值。我想将这些值附加到 DenseVector 并创建一列，但我不确定如何开始。此所需输出的任何提示:

+---------------+ 
| inputsMerged  |  
+---------------+ 
| [1,0,0,12]    |  
+---------------+ 
| [0,1,0,10]    | 
+---------------+ 
... 

编辑:我正在尝试使用 VectorAssembler 方法，但我得到的数组不是预期的。

请您参考如下方法:

你可能会这样做:

df.show() 
+-------------+----+ 
|       inputs|temp| 
+-------------+----+ 
|[1.0,0.0,0.0]|  12| 
|[0.0,1.0,0.0]|  10| 
+-------------+----+ 
 
df.printSchema() 
root 
 |-- inputs: vector (nullable = true) 
 |-- temp: long (nullable = true) 

导入:

import pyspark.sql.functions as F 
from pyspark.ml.linalg import Vectors, VectorUDT 

创建 udf 以合并 Vector 和元素:

concat = F.udf(lambda v, e: Vectors.dense(list(v) + [e]), VectorUDT()) 

将 udf 应用于输入和临时列:

merged_df = df.select(concat(df.inputs, df.temp).alias('inputsMerged')) 
 
merged_df.show() 
+------------------+ 
|      inputsMerged| 
+------------------+ 
|[1.0,0.0,0.0,12.0]| 
|[0.0,1.0,0.0,10.0]| 
+------------------+ 
 
merged_df.printSchema() 
root 
 |-- inputsMerged: vector (nullable = true)