python之使用 scrapy 在 url 中抓取

我正在尝试使用 scrapy 抓取 craigslist 并已成功获取 url，但现在我想从 url 中的页面中提取数据。以下是代码:

from scrapy.spider import BaseSpider 
from scrapy.selector import HtmlXPathSelector 
from craigslist.items import CraigslistItem 
 
class craigslist_spider(BaseSpider): 
    name = "craigslist_unique" 
    allowed_domains = ["craiglist.org"] 
    start_urls = [ 
        "http://sfbay.craigslist.org/search/sof?zoomToPosting=&query=&srchType=A&addFour=part-time", 
        "http://newyork.craigslist.org/search/sof?zoomToPosting=&query=&srchType=A&addThree=internship", 
    "http://seattle.craigslist.org/search/sof?zoomToPosting=&query=&srchType=A&addFour=part-time" 
    ] 
 
 
def parse(self, response): 
   hxs = HtmlXPathSelector(response) 
   sites = hxs.select("//span[@class='pl']") 
   items = [] 
   for site in sites: 
       item = CraigslistItem() 
       item['title'] = site.select('a/text()').extract() 
       item['link'] = site.select('a/@href').extract() 
   #item['desc'] = site.select('text()').extract() 
       items.append(item) 
   hxs = HtmlXPathSelector(response) 
   #print title, link         
   return items 

我是 scrapy 的新手，无法弄清楚如何实际点击 url (href) 并在该 url 的页面内获取数据并对所有 url 执行此操作。

请您参考如下方法:

start_urls的响应在parse方法中一一接收

如果您只想从 start_urls 响应中获取信息，您的代码几乎没问题。但是您的解析方法应该在您的 craigslist_spider 类中，而不是在该类之外。

def parse(self, response): 
   hxs = HtmlXPathSelector(response) 
   sites = hxs.select("//span[@class='pl']") 
   items = [] 
   for site in sites: 
       item = CraigslistItem() 
       item['title'] = site.select('a/text()').extract() 
       item['link'] = site.select('a/@href').extract() 
       items.append(item) 
   #print title, link 
   return items 

如果您想从 start_urls 和 start_urls 响应中出现的 anchor 获取一半信息怎么办？

def parse(self, response): 
    hxs = HtmlXPathSelector(response) 
    sites = hxs.select("//span[@class='pl']") 
    for site in sites: 
        item = CraigslistItem() 
        item['title'] = site.select('a/text()').extract() 
        item['link'] = site.select('a/@href').extract() 
        if item['link']: 
            if 'http://' not in item['link']: 
                item['link'] = urljoin(response.url, item['link']) 
            yield Request(item['link'], 
                          meta={'item': item}, 
                          callback=self.anchor_page) 
 
 
def anchor_page(self, response): 
    hxs = HtmlXPathSelector(response) 
    old_item = response.request.meta['item'] # Receiving parse Method item that was in Request meta 
    # parse some more values 
    #place them in old_item 
    #e.g 
    old_item['bla_bla']=hxs.select("bla bla").extract() 
    yield old_item 

您只需要在解析方法中yield Request 并使用Request 的meta 传送您的旧项目/p>

然后在 anchor_page 中提取 old_item，在其中添加新值并简单地生成它。