Skip to main content
 首页 » 编程设计

python之Scrapy:在 scrapyd 运行多个蜘蛛之python 逻辑错误

2025年01月19日7kuangbin

抓取 1.4

我正在使用这个脚本 ( Run multiple scrapy spiders at once using scrapyd ) 在 Scrapyd 安排多个蜘蛛。在我使用 Scrapy 0.19 之前运行良好。

我收到错误:TypeError: create_crawler() takes exactly 2 arguments (1 given)

所以现在我不知道问题是在 Scrapy 版本还是一个简单的 python 逻辑问题(我是 python 新手)

我做了一些修改以检查蜘蛛是否在数据库上处于事件状态。

class AllCrawlCommand(ScrapyCommand): 
 
    requires_project = True 
    default_settings = {'LOG_ENABLED': False} 
 
    def short_desc(self): 
        return "Schedule a run for all available spiders" 
 
    def run(self, args, opts): 
 
        cursor = get_db_connection() 
        cursor.execute("SELECT * FROM lojas WHERE disponivel = 'S'") 
        rows = cursor.fetchall() 
 
        # Coloco todos os dominios dos sites em uma lista 
        # La embaixo faco uma verificacao para rodar somente os 
        # que estao disponiveis e somente os que batem o dominio do site 
        sites = [] 
        for row in rows: 
            site = row[2] 
            print site 
 
            # adiciono cada site na lista  
            sites.append(site) 
 
        url = 'http://localhost:6800/schedule.json' 
        crawler = self.crawler_process.create_crawler() 
        crawler.spiders.list() 
        for s in crawler.spiders.list(): 
            #print s 
            if s in sites: 
 
                values = {'project' : 'esportifique', 'spider' : s} 
                r = requests.post(url, data=values) 
                print(r.text) 

请您参考如下方法:

根据 parik 建议的链接,这是我所做的:

from scrapy.utils.project import get_project_settings 
from scrapy.crawler import CrawlerProcess 
import requests 
 
setting = get_project_settings() 
process = CrawlerProcess(setting) 
 
url = 'http://localhost:6800/schedule.json' 
 
cursor = get_db_connection() 
cursor.execute("SELECT * FROM lojas WHERE disponivel = 'S'") 
rows = cursor.fetchall() 
 
# Coloco todos os dominios dos sites em uma lista 
# La embaixo faco uma verificacao para rodar somente os 
# que estao disponiveis e somente os que batem o dominio do site 
sites = [] 
for row in rows: 
    site = row[2] 
    print site 
 
    # adiciono cada site na lista  
    sites.append(site) 
 
for spider_name in process.spiders.list(): 
    print ("Running spider %s" % (spider_name)) 
    #process.crawl(spider_name,query="dvh") #query dvh is custom argument used in your scrapy 
    if spider_name in sites: 
        values = {'project' : 'esportifique', 'spider' : spider_name} 
        r = requests.post(url, data=values)