我正在尝试使用 pyspark.ml 计算分类器的精度、召回率和 F1:
model = completePipeline.fit(training)
predictions = model.transform(test)
mm = MulticlassMetrics(predictions.select(["label", "prediction"]).rdd)
labels = sorted(predictions.select("prediction").rdd.distinct().map(lambda r: r[0]).collect())
for label in labels:
print labels
print "Precision = %s" % mm.precision(label=label)
print "Recall = %s" % mm.recall(label=label)
print "F1 Score = %s" % mm.fMeasure(label=label)
metrics = pandas.DataFrame([(label, mm.precision(label=label), mm.recall(label=label), mm.fMeasure(label=label)) for label in labels],
columns=["Precision", "Recall", "F1"])
结果数据框的模式 predictions:
[('features', 'vector'), ('label', 'int'), ('rawPrediction', 'vector'), ('probability', 'vector'), ('prediction', 'double')]
调用mm.precision触发的错误信息:
Traceback (most recent call last):
File "ml_pipeline_factory_test", line 1, in <module>
File "ml_pipeline_factory_test", line 92, in ml_pipeline_factory_test
File "/tmp/conda-9a013169-8b21-43cb-bcbe-06fc31523d3e/real/envs/conda-env/lib/python2.7/site-packages/pyspark/mllib/evaluation.py", line 240, in precision
return self.call("precision", float(label))
File "/tmp/conda-9a013169-8b21-43cb-bcbe-06fc31523d3e/real/envs/conda-env/lib/python2.7/site-packages/pyspark/mllib/common.py", line 146, in call
return callJavaFunc(self._sc, getattr(self._java_model, name), *a)
File "/tmp/conda-9a013169-8b21-43cb-bcbe-06fc31523d3e/real/envs/conda-env/lib/python2.7/site-packages/pyspark/mllib/common.py", line 123, in callJavaFunc
return _java2py(sc, func(*args))
File "/tmp/conda-9a013169-8b21-43cb-bcbe-06fc31523d3e/real/envs/conda-env/lib/python2.7/site-packages/py4j/java_gateway.py", line 1160, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/tmp/conda-9a013169-8b21-43cb-bcbe-06fc31523d3e/real/envs/conda-env/lib/python2.7/site-packages/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/tmp/conda-9a013169-8b21-43cb-bcbe-06fc31523d3e/real/envs/conda-env/lib/python2.7/site-packages/py4j/protocol.py", line 320, in get_return_value
format(target_id, ".", name), value)
Py4JJavaError: An error occurred while calling o371.precision.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 7 in stage 22.0 failed 4 times, most recent failure: Lost task 7.3 in stage 22.0 (TID 153, dhbpdn12.de.t-internal.com, executor 4): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/tmp/conda-e6ac7105-4788-4b4c-9163-ba8763f29ead/real/envs/conda-env/lib/python2.7/site-packages/pyspark/worker.py", line 245, in main
process()
File "/tmp/conda-e6ac7105-4788-4b4c-9163-ba8763f29ead/real/envs/conda-env/lib/python2.7/site-packages/pyspark/worker.py", line 240, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/tmp/conda-e6ac7105-4788-4b4c-9163-ba8763f29ead/real/envs/conda-env/lib/python2.7/site-packages/pyspark/serializers.py", line 372, in dump_stream
vs = list(itertools.islice(iterator, batch))
File "/tmp/conda-9a013169-8b21-43cb-bcbe-06fc31523d3e/real/envs/conda-env/lib/python2.7/site-packages/pyspark/sql/session.py", line 677, in prepare
File "/tmp/conda-9a013169-8b21-43cb-bcbe-06fc31523d3e/real/envs/conda-env/lib/python2.7/site-packages/pyspark/sql/types.py", line 1421, in verify
File "/tmp/conda-9a013169-8b21-43cb-bcbe-06fc31523d3e/real/envs/conda-env/lib/python2.7/site-packages/pyspark/sql/types.py", line 1402, in verify_struct
File "/tmp/conda-9a013169-8b21-43cb-bcbe-06fc31523d3e/real/envs/conda-env/lib/python2.7/site-packages/pyspark/sql/types.py", line 1421, in verify
File "/tmp/conda-9a013169-8b21-43cb-bcbe-06fc31523d3e/real/envs/conda-env/lib/python2.7/site-packages/pyspark/sql/types.py", line 1415, in verify_default
File "/tmp/conda-9a013169-8b21-43cb-bcbe-06fc31523d3e/real/envs/conda-env/lib/python2.7/site-packages/pyspark/sql/types.py", line 1310, in verify_acceptable_types
TypeError: field prediction: DoubleType can not accept object 0 in type <type 'int'>
请您参考如下方法:
如错误信息所示:
TypeError: field prediction: DoubleType can not accept object 0 in type <type 'int'>
类型问题。虽然 int 和 float 在 Python 中通常可以互换，但在 Java 中则不然。
最简单的解决方案是将 label 字段转换到上游:
predictions = (predictions
.withColumn("label", predictions["label"].cast("double")))
