Problem Statement:
解析后,我将每个 URL 发送到 parse_links 以从中提取电子邮件地址。
解析后,如果我从该链接找到电子邮件地址并返回结果,我想停止迭代。
即
In loop suppose there are 2 URLs: example.com/contact and example.com/about
If email address found from example.com/contact then I don't want to scrap the second one. But I'm getting the email address from all links.
这是我的代码:
def parse(self, response):
urls = [
instance.url for instance in LinkExtractor(
allow_domains='example.com'
).extract_links(response)
]
for url in sorted(urls, reverse=True):
request = Request(url, callback=self.parse_links)
yield request
def parse_links(self, response):
item = EmailScraperItem()
mailrex = '[\w\.-]+@[\w\.-]+'
result = response.xpath('//a[@href]').re('%s' % mailrex)
if result:
item['emails'] = result # here how can I send first value and ignore other results
return item
运行爬虫后我得到这个输出:
2017-01-30 20:31:27 [scrapy.core.scraper] DEBUG: Scraped from <200 http://example.com/contact/>
{'emails': ['abc@example.com']} # first result
2017-01-30 20:31:29 [scrapy.core.scraper] DEBUG: Scraped from <200 http://example.com/about/>
{'emails': ['xyz@example.com']} # second result
我只想要第一个。
请您参考如下方法:
由于 Scrapy 的异步特性,你甚至不能确定响应会按照它们发出的顺序到达回调。你可以做的是获取 urls 列表,用 meta 传递它,然后像这样顺序访问 urls:
def parse(self, response):
urls = [
instance.url for instance in LinkExtractor(
allow_domains='example.com'
).extract_links(response)
]
try:
# take url and pass remaining to the callback
return Request(urls.pop(), callback=self.parse_links, meta={'urls': urls})
except IndexError:
pass
def parse_links(self, response):
item = EmailScraperItem()
mailrex = '[\w\.-]+@[\w\.-]+'
result = response.xpath('//a[@href]').re('%s' % mailrex)
if result:
item['emails'] = result # here how can I send first value and ignore other results
return item
# if no emails found, request next url from list
try:
urls = response.meta['urls']
return Request(urls.pop(), callback=self.parse_links, meta={'urls': urls})
except IndexError:
pass
