Skip to main content
 首页 » 编程设计

python之使用 scrapy 链接请求

2024年10月25日17fff_TT

现在我可以看到 scrapy 同时下载所有页面,但我需要的是链接 people 和 extract_person 方法,这样当我在方法 people 我关注他们所有人并抓取我需要的所有信息,然后我才继续使用另一个页面 people urls。我该怎么做?

def people(self, response): 
    sel = Selector(response) 
    urls = sel.xpath(XPATHS.URLS).extract() 
    for url in urls: 
        yield Request( 
            url=BASE_URL+url, 
            callback=self.extract_person, 
        ) 
 
def extract_person(self, response): 
    sel = Selector(response) 
    name = sel.xpath(XPATHS.NAME).extract()[0] 
    person = PersonItem(name=name) 
    yield student 

请您参考如下方法:

您可以控制 priority请求数:

priority (int) – the priority of this request (defaults to 0). The priority is used by the scheduler to define the order used to process requests. Requests with a higher priority value will execute earlier. Negative values are allowed in order to indicate relatively low-priority.

将人员请求的优先级设置为 1 将让 Scrapy 知道首先处理它们:

for url in student_urls: 
    yield Request( 
        url=BASE_URL+url, 
        callback=self.extract_person, 
        priority=1 
    )