Skip to main content
 首页 » 编程设计

python之在 scrapy 中返回第一个遇到的结果

2025年02月15日12小虾米

Problem Statement:

解析后,我将每个 URL 发送到 parse_links 以从中提取电子邮件地址。

解析后,如果我从该链接找到电子邮件地址并返回结果,我想停止迭代。

即

In loop suppose there are 2 URLs: example.com/contact and example.com/about

If email address found from example.com/contact then I don't want to scrap the second one. But I'm getting the email address from all links.

这是我的代码:

def parse(self, response): 
    urls = [ 
        instance.url for instance in LinkExtractor( 
            allow_domains='example.com' 
        ).extract_links(response) 
    ] 
 
    for url in sorted(urls, reverse=True): 
        request = Request(url, callback=self.parse_links) 
        yield request 
 
def parse_links(self, response): 
    item = EmailScraperItem() 
    mailrex = '[\w\.-]+@[\w\.-]+' 
    result = response.xpath('//a[@href]').re('%s' % mailrex) 
    if result: 
        item['emails'] = result  # here how can I send first value and ignore other results 
    return item 

运行爬虫后我得到这个输出:

2017-01-30 20:31:27 [scrapy.core.scraper] DEBUG: Scraped from <200 http://example.com/contact/> 
{'emails': ['abc@example.com']}  # first result 
 
2017-01-30 20:31:29 [scrapy.core.scraper] DEBUG: Scraped from <200 http://example.com/about/> 
{'emails': ['xyz@example.com']}  # second result 

我只想要第一个。

请您参考如下方法:

由于 Scrapy 的异步特性,你甚至不能确定响应会按照它们发出的顺序到达回调。你可以做的是获取 urls 列表,用 meta 传递它,然后像这样顺序访问 urls:

def parse(self, response): 
    urls = [ 
        instance.url for instance in LinkExtractor( 
            allow_domains='example.com' 
        ).extract_links(response) 
    ] 
 
    try: 
       # take url and pass remaining to the callback 
       return Request(urls.pop(), callback=self.parse_links, meta={'urls': urls}) 
    except IndexError: 
       pass 
 
def parse_links(self, response): 
    item = EmailScraperItem() 
    mailrex = '[\w\.-]+@[\w\.-]+' 
    result = response.xpath('//a[@href]').re('%s' % mailrex) 
    if result: 
        item['emails'] = result  # here how can I send first value and ignore other results 
        return item 
    # if no emails found, request next url from list 
    try: 
       urls = response.meta['urls'] 
       return Request(urls.pop(), callback=self.parse_links, meta={'urls': urls}) 
    except IndexError: 
       pass