Skip to main content
 首页 » 编程设计

python之Scrapy Crawler 不跟踪链接

2024年12月31日7dyllove98

我正在编写一个 Scrapy 爬虫来从房地产网站上抓取信息,https://www.iproperty.com.sg/sale/?page=1,https://www.iproperty.com.sg/sale/?page=2 等。这个想法是,对于每一行,从该行获取信息并请求该行上的链接以获取更多信息。处理完该页面上的所有行后,转到下一页并重复:

import scrapy 
from scrapy.linkextractors import LinkExtractor 
from scrapy.spiders import CrawlSpider, Rule 
from property.items import PropertyItem 
 
 
class IpropCrawlerSpider(CrawlSpider): 
    name = 'iprop_crawler' 
    allowed_domains = ['www.iproperty.com.sg'] 
    start_urls = ["https://www.iproperty.com.sg/sale/?page=1"] 
    rules = ( 
        Rule(LinkExtractor(allow=r'sale\/\?page=[1-9]'),  
         callback='parse_item', follow=True), 
    ) 
 
    def parse_item(self, response): 
 
        prop_list_xpath = '//h3[@class="cgiArp"]' 
 
        for prop in response.xpath(prop_list_xpath): 
            item = PropertyItem() 
            item['name'] = prop.xpath('./a/text()').extract_first() 
            deep_uri = prop.xpath('./a/@href').extract_first() 
            deep_url = 'https://www.iproperty.com.sg' + deep_uri 
            request = scrapy.Request(deep_url, callback=self.parse_per_prop) 
            request.meta['item'] = item 
            yield request 
 
    def parse_per_prop(self, response): 
        item = response.meta['item'] 
        item['price'] = response\ 
             .xpath('//div[@class="property-price duzTnm"]/text()')\ 
             .extract_first() 
        item['address'] = response\ 
             .xpath('//span[@class="property-address sale-default"]/text()')\ 
             .extract_first() 
        item['property_type'] = response\ 
             .xpath('//div[@class="property-attr-propertyType cXGbLS"]' \ 
                    + '/div[2]/text()')\ 
             .extract_first() 
        yield item 

运行此爬虫不会抓取任何数据:

2018-11-09 01:53:58 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: property) 
2018-11-09 01:53:58 [scrapy.utils.log] INFO: Versions: lxml 3.7.2.0, libxml2 2.9.4, cssselect 1.0.0, parsel 1.5.0, w3lib 1.17.0, Twisted 17.1.0, Python 3.6.1 |Anaconda custom (64-bit)| (default, Mar 22 2017, 19:54:23) - [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)], pyOpenSSL 16.2.0 (OpenSSL 1.0.2p  14 Aug 2018), cryptography 1.7.1, Platform Linux-4.18.16-arch1-1-ARCH-x86_64-with-arch 
2018-11-09 01:53:58 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'property', 'DOWNLOAD_DELAY': 1, 'NEWSPIDER_MODULE': 'property.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['property.spiders']} 
2018-11-09 01:53:58 [scrapy.middleware] INFO: Enabled extensions: 
['scrapy.extensions.corestats.CoreStats', 
 'scrapy.extensions.telnet.TelnetConsole', 
 'scrapy.extensions.memusage.MemoryUsage', 
 'scrapy.extensions.logstats.LogStats'] 
2018-11-09 01:53:58 [scrapy.middleware] INFO: Enabled downloader middlewares: 
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware', 
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 
 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 
 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 
2018-11-09 01:53:58 [scrapy.middleware] INFO: Enabled spider middlewares: 
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 
 'scrapy.spidermiddlewares.referer.RefererMiddleware', 
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 
 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 
2018-11-09 01:53:58 [scrapy.middleware] INFO: Enabled item pipelines: 
[] 
2018-11-09 01:53:58 [scrapy.core.engine] INFO: Spider opened 
2018-11-09 01:53:58 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
2018-11-09 01:53:58 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6024 
2018-11-09 01:53:58 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.iproperty.com.sg/robots.txt> (referer: None) 
2018-11-09 01:54:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.iproperty.com.sg/sale/?page=1> (referer: None) 
2018-11-09 01:54:01 [scrapy.core.engine] INFO: Closing spider (finished) 
2018-11-09 01:54:01 [scrapy.statscollectors] INFO: Dumping Scrapy stats: 
{'downloader/request_bytes': 460, 
 'downloader/request_count': 2, 
 'downloader/request_method_count/GET': 2, 
 'downloader/response_bytes': 154841, 
 'downloader/response_count': 2, 
 'downloader/response_status_count/200': 2, 
 'finish_reason': 'finished', 
 'finish_time': datetime.datetime(2018, 11, 8, 17, 54, 1, 224281), 
 'log_count/DEBUG': 3, 
 'log_count/INFO': 7, 
 'memusage/max': 47136768, 
 'memusage/startup': 47136768, 
 'response_received_count': 2, 
 'scheduler/dequeued': 1, 
 'scheduler/dequeued/memory': 1, 
 'scheduler/enqueued': 1, 
 'scheduler/enqueued/memory': 1, 
 'start_time': datetime.datetime(2018, 11, 8, 17, 53, 58, 676635)} 
2018-11-09 01:54:01 [scrapy.core.engine] INFO: Spider closed (finished) 

如果我将 parse_item 更改为 parse_start_url,则只会抓取第一页,但不会访问以下链接:

2018-11-09 02:11:42 [scrapy.statscollectors] INFO: Dumping Scrapy stats: 
{'downloader/request_bytes': 6195, 
 'downloader/request_count': 20, 
 'downloader/request_method_count/GET': 20, 
 'downloader/response_bytes': 2433163, 
 'downloader/response_count': 20, 
 'downloader/response_status_count/200': 20, 
 'finish_reason': 'shutdown', 
 'finish_time': datetime.datetime(2018, 11, 8, 18, 11, 42, 430358), 
 'item_scraped_count': 18, 
 'log_count/DEBUG': 39, 
 'log_count/INFO': 8, 
 'memusage/max': 47132672, 
 'memusage/startup': 47132672, 
 'request_depth_max': 1, 
 'response_received_count': 20, 
 'scheduler/dequeued': 19, 
 'scheduler/dequeued/memory': 19, 
 'scheduler/enqueued': 21, 
 'scheduler/enqueued/memory': 21, 
 'start_time': datetime.datetime(2018, 11, 8, 18, 11, 18, 416991)} 
2018-11-09 02:11:42 [scrapy.core.engine] INFO: Spider closed (shutdown) 

我想就这个问题寻求启发,为什么我无法点击下一页的链接。

请您参考如下方法:

根据 Scrapy documentation 判断,看起来您将对 parse_item 方法的引用传递给规则的 callback 参数。但是,根据文档,此回调对提取的链接进行操作。这不是您想要的,因为您的函数需要 Scrapy Response 才能运行。因此,您应该做的是使用 process_request 参数。在相关说明中,我更改了您的正则表达式,因为您现在使用它的方式仅适用于第 1 到 9 页

rules = ( 
    Rule(LinkExtractor(allow = r'sale\/\?page=[1-9]\d*'),  
     process_request = 'parse_item', follow = True), 
) 

顺便说一句,你可能不应该将 Request 对象返回给 Scrapy,而应该使用 scrapy.Item 和 ItemLoader 来存储您的数据。