python之PySpark 程序抛出错误 "TypeError: Invalid argument, not a string or column"

我有一个名为“new_emp_final_1”的数据框。当我尝试通过从 udf 调用函数 difficulty 从 cookTime 和 prepTime 派生列“difficulty”时，它给我错误。

new_emp_final_1.dtypes 如下 -

[('name', 'string'), ('ingredients', 'string'), ('url', 'string'), ('image', 'string'), ('cookTime', 'string'), ('recipeYield', 'string'), ('datePublished', 'strin 
g'), ('prepTime', 'string'), ('description', 'string')] 

new_emp_final_1.schema 的结果是 -

StructType(List(StructField(name,StringType,true),StructField(ingredients,StringType,true),StructField(url,StringType,true),StructField(image,StringType,true),StructField(cookTime,StringType,true),StructField(recipeYield,StringType,true),StructField(datePublished,StringType,true),StructField(prepTime,StringType,true),StructField(description,StringType,true))) 

代码:

def difficulty(cookTime, prepTime):    
    if not cookTime or not prepTime: 
        return "Unkown" 
 
    total_duration = cookTime + prepTime 
    if total_duration > 3600: 
        return "Hard" 
    elif total_duration > 1800 and total_duration < 3600: 
        return "Medium" 
    elif total_duration < 1800: 
        return "Easy"  
    else:  
        return "Unkown" 
 
func_udf = udf(difficulty, IntegerType()) 
new_emp_final_1 = new_emp_final_1.withColumn("difficulty", func_udf(new_emp_final_1.cookTime, new_emp_final_1.prepTime)) 
new_emp_final_1.show(20,False) 

错误是-

File "/home/raghavcomp32915/mypycode.py", line 56, in <module> func_udf = udf(difficulty, IntegerType()) File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/sql/udf.py", line 186, in wrapper File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/sql/udf.py", line 166, in __call__ File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/sql/column.py", line 66, in _to_seq File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/sql/column.py", line 54, in _to_java_column TypeError: Invalid argument, not a string or column: <function difficulty at 0x7f707e9750c8> of type <type 'function'>. For column literals, use 'lit', 'array', 's truct' or 'create_map' function.

我期望现有数据框 new_emp_final_1 中有一个名为 difficulty 的列，其值为 Hard、Medium、Easy 或 Unknown。

请您参考如下方法:

我在使用 Python 的 sum 时遇到了这个问题，因为它与 Spark 的 SQL sum 有冲突——这是一个真实的例子:

from pyspark.sql.functions import * 

不好。

不言而喻，解决方案要么限制对所需函数的导入，要么导入 pyspark.sql.functions 并为其添加前缀。