Skip to main content
 首页 » 编程设计

python之scrapy 脚本运行时没有(明显的)错误但不抓取数据

2025年01月19日8jpfss

我正在关注 this尝试学习如何使用 scrapy 的教程。我目前正在做第二个教程,Writing Custom Spiders。我创建了项目并按照教程指定编写了 redditbot.py。

import scrapy 
 
class RedditbotSpider(scrapy.Spider): 
name = 'redditbot' 
allowed_domains = ['reddit.com/r/gameofthrones/'] 
start_urls = ['http://www.reddit.com/r/gameofthrones//'] 
 
def parse(self, response): 
    #Extracting the content using css selectors 
    titles = response.css('.title.may_blank::text').extract() 
    votes = response.css('.score.unvoted::text').extract() 
    times = response.css('time::attr(title)').extract() 
    comments = response.css('.comments::text').extract() 
 
    #Display the extracted contect in row fashion 
    for item in zip(titles,votes,times,comments): 
        #Creates a dictionary to store the scraped info 
        scraped_info = { 
            'title' : item[0], 
            'vote' : item[1], 
            'created_at' : item[2], 
            'comments' : item[3], 
        } 
 
        #Yield/give the scraped info to scrapy 
        yield scraped_info 

但是,当我使用

运行程序时
scrapy crawl redditbot 

程序将运行但不会输出任何抓取的数据,正如教程所说的那样。这是我在终端中收到的输出:

evans-mbp:ourfirstscraper evanyamaguchi$ scrapy crawl redditbot 
2018-01-04 13:24:53 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: ourfirstscraper) 
2018-01-04 13:24:53 [scrapy.utils.log] INFO: Versions: lxml 4.1.1.0, libxml2 2.9.7, cssselect 1.0.3, parsel 1.3.1, w3lib 1.18.0, Twisted 17.9.0, Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28) - [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)], pyOpenSSL 17.5.0 (OpenSSL 1.1.0g  2 Nov 2017), cryptography 2.1.4, Platform Darwin-17.3.0-x86_64-i386-64bit 
2018-01-04 13:24:53 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'ourfirstscraper', 'FEED_FORMAT': 'csv', 'FEED_URI': 'reddit.csv', 'NEWSPIDER_MODULE': 'ourfirstscraper.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['ourfirstscraper.spiders']} 
2018-01-04 13:24:53 [scrapy.middleware] INFO: Enabled extensions: 
['scrapy.extensions.corestats.CoreStats', 
 'scrapy.extensions.telnet.TelnetConsole', 
 'scrapy.extensions.memusage.MemoryUsage', 
 'scrapy.extensions.feedexport.FeedExporter', 
 'scrapy.extensions.logstats.LogStats'] 
2018-01-04 13:24:53 [scrapy.middleware] INFO: Enabled downloader middlewares: 
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware', 
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 
 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 
 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 
2018-01-04 13:24:53 [scrapy.middleware] INFO: Enabled spider middlewares: 
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 
 'scrapy.spidermiddlewares.referer.RefererMiddleware', 
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 
 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 
2018-01-04 13:24:53 [scrapy.middleware] INFO: Enabled item pipelines: 
[] 
2018-01-04 13:24:53 [scrapy.core.engine] INFO: Spider opened 
2018-01-04 13:24:53 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
2018-01-04 13:24:53 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023 
2018-01-04 13:24:53 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.reddit.com/robots.txt> from <GET http://www.reddit.com/robots.txt> 
2018-01-04 13:24:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.reddit.com/robots.txt> (referer: None) 
2018-01-04 13:24:53 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.reddit.com/r/gameofthrones//> from <GET http://www.reddit.com/r/gameofthrones//> 
2018-01-04 13:24:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.reddit.com/r/gameofthrones//> (referer: None) 
2018-01-04 13:24:53 [scrapy.core.engine] INFO: Closing spider (finished) 
2018-01-04 13:24:53 [scrapy.statscollectors] INFO: Dumping Scrapy stats: 
{'downloader/request_bytes': 945, 
 'downloader/request_count': 4, 
 'downloader/request_method_count/GET': 4, 
 'downloader/response_bytes': 36092, 
 'downloader/response_count': 4, 
 'downloader/response_status_count/200': 2, 
 'downloader/response_status_count/301': 2, 
 'finish_reason': 'finished', 
 'finish_time': datetime.datetime(2018, 1, 4, 18, 24, 53, 755172), 
 'log_count/DEBUG': 5, 
 'log_count/INFO': 7, 
 'memusage/max': 66359296, 
 'memusage/startup': 66359296, 
 'response_received_count': 2, 
 'scheduler/dequeued': 2, 
 'scheduler/dequeued/memory': 2, 
 'scheduler/enqueued': 2, 
 'scheduler/enqueued/memory': 2, 
 'start_time': datetime.datetime(2018, 1, 4, 18, 24, 53, 205879)} 
2018-01-04 13:24:53 [scrapy.core.engine] INFO: Spider closed (finished) 

我不明白为什么蜘蛛似乎在运行但没有从网站上抓取任何数据。

提前致谢, 埃文

请您参考如下方法:

看来你只是打错了字。
类名是may-blank,而不是may_blank。