Skip to main content
 首页 » 编程设计

python之scrapy错误处理url

2025年02月15日7insus

您好,我是 python 和 scrapy 的新手,我正在尝试编写蜘蛛代码,但在处理起始 url 时找不到错误在哪里或错误的解决方案,不知道它是否是xpath 或其他问题,我发现的大多数线程都在谈论错误的缩进,但这不是我的情况。 代码:

import scrapy 
from scrapy.exceptions import CloseSpider 
 
from scrapy_crawls.items import Vino 
 
 
class BodebocaSpider(scrapy.Spider): 
    name = "Bodeboca" 
    allowed_domains = ["bodeboca.com"] 
    start_urls = ( 
        'http://www.bodeboca.com/vino/espana', 
    ) 
    counter = 1 
    next_url = "" 
 
    vino = None 
 
    def __init__(self): 
        self.next_url = self.start_urls[0] 
 
 
    def parse(self, response): 
 
        for sel in response.xpath( 
                '//div[@id="venta-main-wrapper"]/div[@id="venta-main"]/div/div/div/div/div/div/span'): 
 
            #print sel 
            # HREF 
            a_href = sel.xpath('.//a/@href').extract() 
            the_href = a_href[0] 
            print the_href 
            yield scrapy.Request(the_href, callback=self.parse_item, headers={'Referer': response.url.encode('utf-8'), 
                                                                              'Accept-Language': 'es-ES,es;q=0.8,en-US;q=0.5,en;q=0.3'}) 
 
        # SIGUIENTE URL 
        results = response.xpath( 
            '//div[@id="wrapper"]/article/div[@id="article-inner"]/div[@id="default-filter-form-wrapper"]/div[@id="venta-main-wrapper"]/div[@class="bb-product-info-sort bb-sort-behavior-attached"]/div[@clsas="bb-product-info"]/span[@class="bb-product-info-count"]').extract() 
 
 
        if not results: 
            raise CloseSpider 
        else: 
            #self.next_url = self.next_url.replace(str(self.counter), str(self.counter + 1)) 
            #self.counter += 1 
            self.next_url = response.xpath('//div[@id="venta-main-wrapper"]/div[@class="item-list"]/ul[@class="pager"]/li[@class="pager-next"]/a/@href').extract()[0] 
            yield scrapy.Request(self.next_url, callback=self.parse, headers={'Referer': self.allowed_domains[0], 
                                                                              'Accept-Language': 'es-ES,es;q=0.8,en-US;q=0.5,en;q=0.3'}) 

错误:

2017-03-28 12:29:08 [scrapy.core.engine] INFO: Spider opened 
2017-03-28 12:29:08 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
2017-03-28 12:29:08 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.bodeboca.com/robots.txt> (referer: None) 
2017-03-28 12:29:08 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.bodeboca.com/vino/espana> (referer: None) 
/vino/terra-cuques-2014 
2017-03-28 12:29:08 [scrapy.core.scraper] ERROR: Spider error processing <GET http://www.bodeboca.com/vino/espana> (referer: None) 
 
Traceback (most recent call last): 
  File "/usr/local/lib/python2.7/dist-packages/scrapy/utils/defer.py", line 102, in iter_errback 
yield next(it) 
    File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output 
for x in result: 
  File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/referer.py", line 22, in <genexpr> 
return (_set_referer(r) for r in result or ()) 
  File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr> 
return (r for r in result or () if _filter(r)) 
  File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr> 
return (r for r in result or () if _filter(r)) 
  File "/home/gerardo/proyectos/vinos-diferentes-crawl/scrapy_crawls/spiders/Bodeboca.py", line 36, in parse 
'Accept-Language': 'es-ES,es;q=0.8,en-US;q=0.5,en;q=0.3'}) 
  File "/usr/local/lib/python2.7/dist-packages/scrapy/http/request/__init__.py", line 25, in __init__ 
self._set_url(url) 
  File "/usr/local/lib/python2.7/dist-packages/scrapy/http/request/__init__.py", line 57, in _set_url 
raise ValueError('Missing scheme in request url: %s' % self._url) 
ValueError: Missing scheme in request url: /vino/terra-cuques-2014 
2017-03-28 12:29:08 [scrapy.core.engine] INFO: Closing spider (finished) 
2017-03-28 12:29:08 [scrapy.statscollectors] INFO: Dumping Scrapy stats: 
{'downloader/request_bytes': 449, 
 'downloader/request_count': 2, 
 'downloader/request_method_count/GET': 2, 
 'downloader/response_bytes': 38558, 
 'downloader/response_count': 2, 
 'downloader/response_status_count/200': 2, 
 'finish_reason': 'finished', 
 'finish_time': datetime.datetime(2017, 3, 28, 10, 29, 8, 951654), 
 'log_count/DEBUG': 2, 
 'log_count/ERROR': 1, 
 'log_count/INFO': 7, 
 'response_received_count': 2, 
 'scheduler/dequeued': 1, 
 'scheduler/dequeued/memory': 1, 
 'scheduler/enqueued': 1, 
 'scheduler/enqueued/memory': 1, 
 'spider_exceptions/ValueError': 1, 
 'start_time': datetime.datetime(2017, 3, 28, 10, 29, 8, 690948)} 
2017-03-28 12:29:08 [scrapy.core.engine] INFO: Spider closed (finished) 

请您参考如下方法:

简单回答:您从页面中提取相对 url,例如 /vino/terra-cuques-2014

为了使 scrapy 请求 url 需要完整: http://www.bodeboca.com/vino/terra-cuques-2014。 你可以使用 Scrapy 制作完整的 url response.urljoin() 方法例如: full_url = response.urljoin(url)。

尽量不要使用像这样的 xpath 表达式:/div[@id="venta-main"]/div/div/div/div/div/div/span - 它很难阅读并且页面的最细微变化很容易破坏。相反,您可以简单地使用基于类的 xpath://a[@class="verficha"]。

你可以像这样重写你的蜘蛛的一部分:

def parse(self, response): 
    links = response.xpath('//a[@class="verficha"]') 
    for link in links: 
        url = link.xpath('@href').extract_first() 
        full_url = response.urljoin(url) 
        yield scrapy.Request(full_url , callback= your callback) 

如果你想提取 url 到下一页,你可以使用 xpath next_page = response.xpath('//li[@class="pager-next"]/a/@href').extract_first() ,再次调用response.urljoin(next_page)等