Skip to main content
 首页 » 编程设计

python之Scrapy 只抓取 15 个起始 url 列表中的第一个起始 url

2024年10月24日13lexus

我是 Scrapy 的新手,正在尝试自学基础知识。我编写了一个代码,可以访问路易斯安那州自然资源部网站,以检索某些油井的序列号。

我在开始 URL 命令中列出了每个井的链接,但 scrappy 只从第一个 url 下载数据。我做错了什么?

import scrapy 
from scrapy import Spider 
from scrapy.selector import Selector 
from mike.items import MikeItem 
 
class SonrisSpider(Spider): 
   name = "sspider" 
 
   start_urls = [ 
    "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=207899", 
    "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=971683", 
    "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=214206", 
    "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=159420", 
    "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=243671", 
    "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=248942", 
    "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=156613", 
    "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=972498", 
    "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=215443", 
    "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=248463", 
    "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=195136", 
    "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=179181", 
    "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=199930", 
    "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=203419", 
    "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=220454", 
 
] 
 
 
    def parse(self, response): 
       item = MikeItem() 
       item['serial'] =    response.xpath('/html/body/table[1]/tr[2]/td[1]/text()').extract()[0] 
       yield item    

感谢您提供的任何帮助。如果我没有彻底解释我的问题,请告诉我,我会尝试澄清。

请您参考如下方法:

我认为这段代码可能会有所帮助,

默认情况下,scrapy 会阻止重复请求。由于您的起始网址中只有参数不同,scrapy 会将起始网址中的其余网址视为第一个网址的重复请求。这就是为什么您的蜘蛛在获取第一个 url 后停止的原因。为了解析其余的 url,我们在 scrapy 请求中启用了 dont_filter 标志。 (查看 start_request())

# -*- coding: utf-8 -*- 
import scrapy 
from scrapy.http import Request 
from mike.items import MikeItem 
 
 
class SonrisSpider(scrapy.Spider): 
    name = "sspider" 
    allowed_domains = ["sonlite.dnr.state.la.us"] 
    start_urls = [ 
                "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=207899", 
                "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=971683", 
                "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=214206", 
                "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=159420", 
                "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=243671", 
                "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=248942", 
                "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=156613", 
                "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=972498", 
                "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=215443", 
                "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=248463", 
                "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=195136", 
                "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=179181", 
                "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=199930", 
                "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=203419", 
                "http://sonlite.dnr.state.la.us/sundown/cart_prod/cart_con_wellinfo2?p_WSN=220454", 
            ] 
 
    def start_requests(self): 
        for url in self.start_urls: 
            yield Request(url=url, callback=self.parse_data, dont_filter=True) 
 
    def parse_data(self, response): 
        item = MikeItem() 
        serial = response.xpath( 
            '/html/body/table[1]/tr[2]/td[1]/text()').extract() 
        serial = serial[0] if serial else 'n/a' 
        item['serial'] = serial 
        yield item 

这个蜘蛛返回的示例输出如下,

{'serial': u'207899'} 
{'serial': u'971683'} 
{'serial': u'214206'} 
{'serial': u'159420'} 
{'serial': u'248942'} 
{'serial': u'243671'}