Skip to main content
 首页 » 编程设计

python之Scrapy 蜘蛛 :relative link and absolute link

2024年10月24日15dudu

在Scrapy Documentation Release 1.0.3中有一个例子,第7行,urljoin方法在链接是相对链接时使用,当链接是绝对链接时,我该怎么办?

示例代码:

import scrapy 
 
 
class StackOverflowSpider(scrapy.Spider): 
    name = 'stackoverflow' 
    start_urls = ['http://stackoverflow.com/questions?sort=votes'] 
 
    def parse(self, response): 
        for href in response.css('.question-summary h3 a::attr(href)'): 
            full_url = response.urljoin(href.extract()) 
            yield scrapy.Request(full_url, callback=self.parse_question) 
 
    def parse_question(self, response): 
        yield { 
            'title': response.css('h1 a::text').extract()[0], 
            'votes': response.css('.question .vote-count-post::text').extract()[0], 
            'body': response.css('.question .post-text').extract()[0], 
            'tags': response.css('.question .post-tag::text').extract(), 
            'link': response.url, 
        } 

请您参考如下方法:

您不必担心,urljoin() 会正确处理这两种情况:

In [1]: response.urljoin("http://stackoverflow.com/questions/426258/checking-a-checkbox-with-jquery") 
Out[1]: 'http://stackoverflow.com/questions/426258/checking-a-checkbox-with-jquery' 
 
In [2]: response.urljoin("/questions/426258/checking-a-checkbox-with-jquery") 
Out[2]: 'http://stackoverflow.com/questions/426258/checking-a-checkbox-with-jquery'