Skip to main content
 首页 » 编程设计

python之Scrapy:所有的爬虫都失败了。有语法错误的爬虫

2025年01月19日10bhlsheji

有时,当一个爬虫中存在一些错误时,我会遇到所有爬虫都失败的情况。 例子: 我有语法错误的爬虫被遗漏了。

class MySpiderWithSyntaxError(scrapy.Spider): 
    name = "my_spider_with_syntax_error" 
 
    start_urls = [ 
        'http://www.website.com' 
    ] 
 
    def parse(self response): 
        for url in response.css('a.p::attr(href)').extract(): 
            print url 

在这个蜘蛛中错过了行中的逗号

def parse(self response): 

蜘蛛 MySpiderWithSyntaxError 将失败。 但是如果运行另一个没有语法错误的蜘蛛(下面的蜘蛛代码)

class MySpiderWithoutSyntaxError(scrapy.Spider): 
    name = "my_spider_without_syntax_error" 
 
    start_urls = [ 
        'http://www.website.com' 
    ] 
 
    def parse(self, response): 
        for url in response.css('a.p::attr(href)').extract(): 
            print url 

我得到这样的错误:

    Traceback (most recent call last): 
    File "/home/Documents/project/.env/bin/scrapy", line 11, in <module> 
       sys.exit(execute()) 
    File "/home/Documents/project/.env/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 141, in execute 
       cmd.crawler_process = CrawlerProcess(settings) 
    File "/home/Documents/project/.env/local/lib/python2.7/site-packages/scrapy/crawler.py", line 238, in __init__ 
       super(CrawlerProcess, self).__init__(settings) 
    File "/home/Documents/project/.env/local/lib/python2.7/site-packages/scrapy/crawler.py", line 129, in __init__ 
       self.spider_loader = _get_spider_loader(settings) 
    File "/home/Documents/project/.env/local/lib/python2.7/site-packages/scrapy/crawler.py", line 325, in _get_spider_loader 
       return loader_cls.from_settings(settings.frozencopy()) 
    File "/home/Documents/project/.env/local/lib/python2.7/site-packages/scrapy/spiderloader.py", line 33, in from_settings 
       return cls(settings) 
    File "/home/Documents/project/.env/local/lib/python2.7/site-packages/scrapy/spiderloader.py", line 20, in __init__ 
       self._load_all_spiders() 
    File "/home/Documents/project/.env/local/lib/python2.7/site-packages/scrapy/spiderloader.py", line 28, in _load_all_spiders 
       for module in walk_modules(name): 
    File "/home/Documents/project/.env/local/lib/python2.7/site-packages/scrapy/utils/misc.py", line 71, in walk_modules 
       submod = import_module(fullpath) 
    File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module 
       __import__(name) 
    File "/home/Documents/project/scrapers/scrapy/spiders/my_spider_with_syntax_error.py", line 14 
       def parse(self response): 
                     ^ 
    SyntaxError: invalid syntax 

问题: 是否有可能捕获这样的错误并且仅使具有语法错误的蜘蛛失败但另一个蜘蛛工作正常?

请您参考如下方法:

如果您使用 Scrapy 项目,那么即使您运行单个蜘蛛(使用 scrapy crawl <spidername> ),也会加载所有蜘蛛模块。因此,如果其中任何一个包含语法错误,您就会收到错误消息。