我想将本地文件读入 spark。我正在使用 Windows。使用了以下命令:
input = sc.textFile("D://sample.txt")
我尝试了所有可能的组合，但不断出现以下或非常相似的错误。
我尝试了以下方法:
添加 file:///和 file://
添加文件:\\和文件:\
D:/sample.txt
D:\sample.txt
D:\sample.txt
当前工作目录为D:\，文件存在于D:\
有什么想法吗？
>>> input = sc.textFile("D://sample.txt")
15/10/27 02:37:37 INFO MemoryStore: ensureFreeSpace(157288) called with curMem=7
891904, maxMem=556038881
15/10/27 02:37:37 INFO MemoryStore: Block broadcast_46 stored as values in memor
y (estimated size 153.6 KB, free 522.6 MB)
15/10/27 02:37:37 INFO MemoryStore: ensureFreeSpace(14276) called with curMem=80
49192, maxMem=556038881
15/10/27 02:37:37 INFO MemoryStore: Block broadcast_46_piece0 stored as bytes in
memory (estimated size 13.9 KB, free 522.6 MB)
15/10/27 02:37:37 INFO BlockManagerInfo: Added broadcast_46_piece0 in memory on
localhost:52887 (size: 13.9 KB, free: 529.6 MB)
15/10/27 02:37:37 INFO SparkContext: Created broadcast 46 from textFile at null:
-1
另外，我们在使用 Windows 时是否总是在命令行中使用反斜杠？还是仅用于目录？
非常感谢， 科比
请您参考如下方法:
可以给os.path.normpath吗一试
import os
input = sc.textFile(os.path.normpath("D:/sample.txt"))
os.path.normpath(path)
Normalize a pathname by collapsing redundant separators and up-level references so that A//B, A/B/, A/./B and A/foo/../B all become A/B. This string manipulation may change the meaning of a path that contains symbolic links. On Windows, it converts forward slashes to backward slashes. To normalize case, use normcase().
来源:https://docs.python.org/2/library/os.path.html#os.path.normpath
