我正在抓取某个网站。在某些情况下,我可能想请求访问未在 allowed_domains 中列出的网站。那可能吗?如果没有,我可以暂时在其中添加域,创建一个请求,然后从我的解析器回调中删除该域吗?
请您参考如下方法:
在 Request 对象 ( documentation ) 上设置 dont_filter=True:
dont_filter (boolean) – indicates that this request should not be filtered by the scheduler.
例子:
from scrapy.spider import BaseSpider
from scrapy.http import Request
class MySpider(BaseSpider):
name = 'wikipedia'
allowed_domains = ['en.wikipedia.org']
start_urls = [
'http://en.wikipedia.org/wiki/Main_Page',
]
def parse(self, response):
print "I'm at wikipedia"
request = Request(url="https://google.com",
callback=self.parse_google,
dont_filter=True)
yield request
def parse_google(self, response):
print "I'm at google"
