我在使用 spark-submit 运行 python 文件时遇到了问题。当 'map' 代码在 'executor' 中运行时，问题如下:
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 151, in _run_module_as_main
mod_name, loader, code, fname = _get_module_details(mod_name)
File "/usr/lib64/python2.7/runpy.py", line 101, in _get_module_details
loader = get_loader(mod_name)
File "/usr/lib64/python2.7/pkgutil.py", line 464, in get_loader
return find_loader(fullname)
File "/usr/lib64/python2.7/pkgutil.py", line 474, in find_loader
for importer in iter_importers(fullname):
File "/usr/lib64/python2.7/pkgutil.py", line 430, in iter_importers
__import__(pkg)
File "/data8/yarn/local-dir/usercache/bo.feng/appcache/application_1448854352032_70810/container_1448854352032_70810_01_000002/pyspark.zip/pyspark/__init__.py", line 41, in <module>
File "/data8/yarn/local-dir/usercache/bo.feng/appcache/application_1448854352032_70810/container_1448854352032_70810_01_000002/pyspark.zip/pyspark/context.py", line 35, in <module>
File "/data8/yarn/local-dir/usercache/bo.feng/appcache/application_1448854352032_70810/container_1448854352032_70810_01_000002/pyspark.zip/pyspark/rdd.py", line 51, in <module>
File "/data8/yarn/local-dir/usercache/bo.feng/appcache/application_1448854352032_70810/container_1448854352032_70810_01_000002/pyspark.zip/pyspark/shuffle.py", line 33, in <module>
File "build/bdist.linux-x86_64/egg/psutil/__init__.py", line 89, in <module>
File "build/bdist.linux-x86_64/egg/psutil/_pslinux.py", line 24, in <module>
File "build/bdist.linux-x86_64/egg/_psutil_linux.py", line 7, in <module>
File "build/bdist.linux-x86_64/egg/_psutil_linux.py", line 4, in __bootstrap__
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 945, in resource_filename
self, resource_name
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1633, in get_resource_filename
self._extract_resource(manager, self._eager_to_zip(name))
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1661, in _extract_resource
self.egg_name, self._parts(zip_path)
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1025, in get_cache_path
self.extraction_error()
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 991, inextraction_error
raise err
pkg_resources.ExtractionError: Can't extract file(s) to egg cache
The following error occurred while trying to extract file(s) to the Python egg
cache:
[Errno 13] Permission denied: '/home/.python-eggs'
The Python egg cache directory is currently set to:
/home/.python-eggs
Perhaps your account does not have write access to this directory? You can
change the cache directory by setting the PYTHON_EGG_CACHE environment
variable to point to an accessible directory.
我给每个executor都设置了PYTHON_EGG_CACHE环境变量，并且在我的程序中也写了'os.environ['PYTHON_EGG_CACHE'] = "/tmp/"'，但是问题依旧。
我的代码:
import os,sys
print "env::::"+os.environ['PYTHON_EGG_CACHE']
from pyspark import SparkConf, SparkContext
# Load and parse the data
def parsePoint(line):
import os
print "env::::"+os.environ['PYTHON_EGG_CACHE']
os.environ['PYTHON_EGG_CACHE'] = "/tmp/"
values = [float(x) for x in line.split(' ')]
return line
if __name__ == "__main__":
os.environ['PYTHON_EGG_CACHE'] = "/tmp/"
print "env::::"+os.environ['PYTHON_EGG_CACHE']
conf = SparkConf()
sc = SparkContext(conf = conf)
data = sc.textFile(sys.argv[1])
parsedData = data.map(parsePoint)
parsedData.collect()
我在“独立”模式下运行此 python 程序并成功。 这是我的提交命令:
spark-submit --name test_py --master yarn-client testpy.py input/sample_svm_data.txt
是 yarn 的问题吗？
请您参考如下方法:
这已经晚了，但这是我在 google 上发现的第一个结果...之前的答案很有帮助(我想知道我必须修改哪些环境变量)，但请不要修改编辑 Spark 源，只是使用适当的工具更改环境变量，将其添加到您的 spark.conf 变量中...
spark.executorEnv.PYTHON_EGG_CACHE="./.python-eggs/"
spark.executorEnv.PYTHON_EGG_DIR="./.python-eggs/"
spark.driverEnv.PYTHON_EGG_CACHE="./.python-eggs/"
spark.driverEnv.PYTHON_EGG_DIR="./.python-eggs/"
(我不想使用/tmp/因为 . 会在我的工作结束后被删除，所以 eggs 也应该消失 IMO)
