python之在不阻塞进程的情况下启动 scrapy multiple spider

我试图在单独的脚本中执行 scrapy spider，当我循环执行此脚本时(例如使用不同参数运行同一个 spider)，我得到 ReactorAlreadyRunning。我的片段:

from celery import task 
from episode.skywalker.crawlers import settings 
from multiprocessing.queues import Queue 
from scrapy import log, project, signals 
from scrapy.settings import CrawlerSettings 
from scrapy.spider import BaseSpider 
from scrapy.spidermanager import SpiderManager 
from scrapy.xlib.pydispatch import dispatcher 
import multiprocessing 
from twisted.internet.error import ReactorAlreadyRunning 
 
 
class CrawlerWorker(multiprocessing.Process): 
 
    def __init__(self, spider, result_queue): 
        from scrapy.crawler import CrawlerProcess 
        multiprocessing.Process.__init__(self) 
        self.result_queue = result_queue 
        self.crawler = CrawlerProcess(CrawlerSettings(settings)) 
        if not hasattr(project, 'crawler'): 
            self.crawler.install() 
        self.crawler.configure() 
 
        self.items = [] 
        self.spider = spider 
        dispatcher.connect(self._item_passed, signals.item_passed) 
 
    def _item_passed(self, item): 
        self.items.append(item) 
 
    def run(self): 
        self.crawler.crawl(self.spider) 
        try: 
            self.crawler.start() 
        except ReactorAlreadyRunning: 
            pass 
 
        self.crawler.stop() 
        self.result_queue.put(self.items) 
 
 
@task 
def execute_spider(spider, **spider__kwargs): 
    ''' 
    Execute spider within separate process 
    @param spider: spider class to crawl or the name (check if instance) 
    ''' 
 
    if not isinstance(spider, BaseSpider): 
        manager = SpiderManager(settings.SPIDER_MODULES) 
        spider = manager.create(spider, **spider__kwargs) 
    result_queue = Queue() 
    crawler = CrawlerWorker(spider, result_queue) 
    crawler.start() 
    items = [] 
 
    for item in result_queue.get(): 
        items.append(item) 

我的建议是它是由多个扭曲的 react 器运行引起的。 我怎样才能避免它？通常有没有反应器运行蜘蛛的方法？

请您参考如下方法:

我想通了，是什么导致了这个问题:如果您在CrawlerWorker 进程中以某种方式调用execute_spider 方法(例如通过递归)，它导致创建第二个 react 器，这是不可能的。

我的解决方案:在 execute_spider 方法中移动所有导致递归调用的语句，因此它们将在同一进程中触发蜘蛛执行，而不是在辅助 爬虫 worker 。我也内置了这样的语句

try: 
        self.crawler.start() 
except ReactorAlreadyRunning: 
        raise RecursiveSpiderCall("Spider %s was called from another spider recursively. Such behavior is not allowed" % (self.spider)) 

捕获蜘蛛无意中的递归调用。