我有以下源文件。我的文件中有一个名为“john”的名字，想要拆分为列表 ['j','o','h','n']。请按如下方式查找人员文件。
源文件:
id,name,class,start_data,end_date
1,john,xii,20170909,20210909
代码:
from pyspark.sql import SparkSession
def main():
spark = SparkSession.builder.appName("PersonProcessing").getOrCreate()
df = spark.read.csv('person.txt', header=True)
nameList = [x['name'] for x in df.rdd.collect()]
print(list(nameList))
df.show()
if __name__ == '__main__':
main()
实际输出:
[u'john']
期望的输出:
['j','o','h','n']
请您参考如下方法:
如果你想在 python 中:
nameList = [c for x in df.rdd.collect() for c in x['name']]
或者如果你想在 spark 中这样做:
from pyspark.sql import functions as F
df.withColumn('name', F.split(F.col('name'), '')).show()
结果:
+---+--------------+-----+----------+--------+
| id| name|class|start_data|end_date|
+---+--------------+-----+----------+--------+
| 1|[j, o, h, n, ]| xii| 20170909|20210909|
+---+--------------+-----+----------+--------+
