Skip to main content
 首页 » 编程设计

python之使用 Scrapy 在 Craigslist 上递归抓取

2024年11月01日10xing901022

我一直在尝试通过构建 scraper 来磨练我的 python 技能,并且最近从 bs4 切换到 scrapy,以便我可以使用它的多线程和下载延迟功能。我已经能够制作一个基本的抓取器并将数据输出到 csv,但是当我尝试添加递归功能时我遇到了问题。我尝试遵循 Scrapy Recursive download of Content 的建议但不断收到以下错误:

调试:重试 http://medford.craigslist.org%20%5Bu'/cto/4359874426.html'%5D> DNS 查找失败:找不到地址

这让我觉得我尝试加入链接的方式是行不通的,因为它是在 url 中插入字符,但我不知道如何修复它。有什么建议吗?

代码如下:

#------------------------------------------------------------------------------- 
# Name:        module1 
# Purpose: 
# 
# Author:      CD 
# 
# Created:     02/03/2014 
# Copyright:   (c) CD 2014 
# Licence:     <your licence> 
#------------------------------------------------------------------------------- 
from scrapy.spider import BaseSpider 
from scrapy.selector import HtmlXPathSelector 
from craigslist_sample.items import CraigslistSampleItem 
from scrapy.contrib.spiders import CrawlSpider, Rule 
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor 
from scrapy.http import Request 
from scrapy.selector import * 
 
class PageSpider(BaseSpider): 
    name = "cto" 
 
    start_urls = ["http://medford.craigslist.org/cto/"] 
 
    rules = (Rule(SgmlLinkExtractor(allow=("index\d00\.html", ), restrict_xpaths=('//p[@class="nextpage"]' ,)) 
        , callback="parse", follow=True), ) 
 
    def parse(self, response): 
        hxs = HtmlXPathSelector(response) 
        titles = hxs.select("//span[@class='pl']") 
        for titles in titles: 
            item = CraigslistSampleItem() 
            item['title'] = titles.select("a/text()").extract() 
            item['link'] = titles.select("a/@href").extract() 
 
            url = "http://medford.craiglist.org %s" % item['link'] 
            yield Request(url=url, meta={'item': item}, callback=self.parse_item_page) 
 
    def parse_item_page(self, response): 
        hxs = HtmlXPathSelector(response) 
 
        item = response.meta['item'] 
        item['description'] = hxs.select('//section[@id="postingbody"]/text()').extract() 
        return item 

请您参考如下方法:

结果是你的代码:

 url = "http://medford.craiglist.org %s" % item['link'] 

生成:

http://medford.craigslist.org [u'/cto/4359874426.html'] 

item['link'] 在您的代码中返回一个列表,而不是您期望的字符串。你需要这样做:

url = 'http://medford.craiglist.org{}'.format(''.join(item['link']))