from scrapy.spider import BaseSpider
class dmozSpider(BaseSpider):
name = "dmoz"
allowed_domains = ["dmoz.org"]
start_urls = [
"http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
"http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
]
def parse(self, response):
filename = response.url.split("/")[-2]
open(filename, 'wb').write(response.body)
然后我运行“scrapy crawl dmoz” 然后我得到了这个错误:
2013-09-14 13:20:56+0700 [dmoz] DEBUG: Retrying http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/> (failed 1 times): Connection to other side was lost in a non-clean fashion.
有人知道如何解决这个问题吗?
请您参考如下方法:
您需要检查您的互联网连接,或者如果您正在使用代理,请为代理身份验证设置环境变量。
在 Windows 中,尝试以下步骤:
- Win+R 输入“systempropertiesadvanced”(没有 引用)
- 点击“环境变量...”按钮
- 添加 2 个新变量(用户/系统变量都可以):
name | value ------------+-------------------------------- HTTP_PROXY | http://username:password@host:port HTTPS_PROXY | https://username:password@host:port
替代方式:setting-proxy-env
