Skip to main content
 首页 » 编程设计

python之如何使用 Scrapy 递归地从站点中抓取每个链接

2024年12月31日7zfyouxi

我正在尝试使用 Scrapy 从网站获取每个链接(而不是其他数据)。我想通过从主页开始,从那里抓取所有链接,然后对于找到的每个链接,跟踪链接并从该页面抓取所有(唯一)链接,并对找到的所有链接执行此操作,直到没有更多跟随。

我还必须输入用户名和密码才能进入站点上的每个页面,因此我在 start_requests 中包含了一个基本的身份验证组件。

到目前为止,我有一个蜘蛛只给我主页上的链接,但我似乎无法弄清楚为什么它不跟踪链接并抓取其他页面。

这是我的蜘蛛:

    from examplesite.items import ExamplesiteItem 
    import scrapy 
    from scrapy.linkextractor import LinkExtractor 
    from scrapy.spiders import Rule, CrawlSpider 
    from scrapy import Request 
    from w3lib.http import basic_auth_header 
    from scrapy.crawler import CrawlerProcess 
 
    class ExampleSpider(CrawlSpider): 
#name of crawler 
name = "examplesite" 
 
#only scrape on pages within the example.co.uk domain 
allowed_domains = ["example.co.uk"] 
 
#start scraping on the site homepage once credentials have been authenticated 
def start_requests(self): 
    url = str("https://example.co.uk") 
    username = "*********" 
    password = "*********" 
    auth = basic_auth_header(username, password) 
    yield scrapy.Request(url=url,headers={'Authorization': auth}) 
 
#rules for recursively scraping the URLS found 
rules = [ 
    Rule( 
        LinkExtractor( 
            canonicalize=True, 
            unique=True 
        ), 
        follow=True, 
        callback="parse" 
    ) 
] 
 
#method to identify hyperlinks by xpath and extract hyperlinks as scrapy items 
def parse(self, response): 
    for element in response.xpath('//a'): 
        item = ExamplesiteItem() 
        oglink = element.xpath('@href').extract() 
        #need to add on prefix as some hrefs are not full https URLs and thus cannot be followed for scraping 
        if "http" not in str(oglink): 
            item['link'] = "https://example.co.uk" + oglink[0] 
        else: 
            item['link'] = oglink 
 
        yield item 

这是我的项目类:

    from scrapy import Field, Item 
 
    class ExamplesiteItem(Item): 
        link = Field() 

我认为我出错的地方是“规则”,我知道你需要点击链接,但我不完全理解它是如何工作的(已经尝试在线阅读一些解释但仍然没有当然)。

如有任何帮助,我们将不胜感激!

请您参考如下方法:

您的规则没问题,问题是覆盖了 parse 方法。

来自 https://doc.scrapy.org/en/latest/topics/spiders.html#crawling-rules 的 scrapy 文档

When writing crawl spider rules, avoid using parse as callback, since the CrawlSpider uses the parse method itself to implement its logic. So if you override the parse method, the crawl spider will no longer work.