python之在 Jupyter 笔记本中运行 pySpark之Windows

我想从 Jupyter notebook 运行 pySpark。我下载并安装了带有 Juptyer 的 Anaconda。我创建了以下几行

 from pyspark import SparkConf, SparkContext 
 conf = SparkConf().setMaster("local").setAppName("My App") 
 sc = SparkContext(conf = conf) 

出现以下错误

ImportError                               Traceback (most recent call last) 
<ipython-input-3-98c83f0bd5ff> in <module>() 
  ----> 1 from pyspark import SparkConf, SparkContext 
  2 conf = SparkConf().setMaster("local").setAppName("My App") 
  3 sc = SparkContext(conf = conf) 
 
 C:\software\spark\spark-1.6.2-bin-hadoop2.6\python\pyspark\__init__.py in   <module>() 
 39  
 40 from pyspark.conf import SparkConf 
  ---> 41 from pyspark.context import SparkContext 
 42 from pyspark.rdd import RDD 
 43 from pyspark.files import SparkFiles 
 
 C:\software\spark\spark-1.6.2-bin-hadoop2.6\python\pyspark\context.py in <module>() 
 26 from tempfile import NamedTemporaryFile 
 27  
 ---> 28 from pyspark import accumulators 
 29 from pyspark.accumulators import Accumulator 
 30 from pyspark.broadcast import Broadcast 
 
 ImportError: cannot import name accumulators 

我尝试根据 Stackoverflow 中的答案添加以下指向 spark/python 目录的环境变量 PYTHONPATH importing pyspark in python shell

但这并没有帮助

请您参考如下方法:

这对我有用:

import os 
import sys 
 
spark_path = "D:\spark" 
 
os.environ['SPARK_HOME'] = spark_path 
os.environ['HADOOP_HOME'] = spark_path 
 
sys.path.append(spark_path + "/bin") 
sys.path.append(spark_path + "/python") 
sys.path.append(spark_path + "/python/pyspark/") 
sys.path.append(spark_path + "/python/lib") 
sys.path.append(spark_path + "/python/lib/pyspark.zip") 
sys.path.append(spark_path + "/python/lib/py4j-0.9-src.zip") 
 
from pyspark import SparkContext 
from pyspark import SparkConf 
 
sc = SparkContext("local", "test") 

验证:

In [2]: sc 
Out[2]: <pyspark.context.SparkContext at 0x707ccf8>