python之使用 Scrapy 爬取带有 JavaScript 的站点

我需要爬取this website .这是一个脑筋急转弯网站，当你点击一个按钮时，它会运行一个 JavaScript 来显示答案窗口。

 <tr>  
      <td width="60" bgcolor="#ECF5FF"> <p align="center"><font color="#800000".htm>1</font></p></td> 
      <td width="539" bgcolor="#ECF5FF">&nbsp;<font color="#008080">一种东西，东方人的短，西方人的长，结婚后女的就可以用男的这东西，和尚有但是不用它&nbsp;</font>  
      </td> 
      <td width="95" bgcolor="#ECF5FF"> <p align="center">  
          <INPUT onClick="MM_popupMsg('答案：名字&nbsp;')" type=button value=答案 name=button8639 style='font-size:12px;height:18px;border:1px solid black;'> 
        </p></td> 
    </tr> 

这是我写的爬取问题和答案的代码。我可以成功得到问题，但未能得到答案。 (当我打印出答案时，它是一个空的 []。)

    questions = hxs.select('//td[@width="539"]/font/text()').extract() 
    answers = hxs.select('//td[@width="95"]/INPUT/@onClick').extract() 

答案是onclick脚本的内容，即:我想得到这个字符串:

MM_popupMsg('答案：名字 ') 

这是我的蜘蛛:

from scrapy.spider import BaseSpider 
from scrapy.selector import HtmlXPathSelector 
import re 
 
class ReviewSpider(BaseSpider): 
    name = "2345jzw" 
 
    allowed_domains = ['2345.com/jzw'] 
    start_urls = ['http://www.2345.com/jzw/index.htm'] 
 
    page = 1 
    while page <= 1: 
        url = 'http://www.2345.com/jzw/%d.htm' % page 
        start_urls.append(url) 
        page = page + 1 
 
    def parse(self, response): 
        hxs = HtmlXPathSelector(response) 
        questions = hxs.select('//td[@width="539"]/font/text()').extract() 
        answers = hxs.select('//td[3]/p/INPUT/@onClick').extract() 
 
        print questions 
        print answers 
 
        id = 1 
        while id <= 50: 
 
            question = questions[id - 1] 
 
 
            question = re.sub(r'<[^>]*?>', '', str(question.encode('utf8'))) 
            question = ' '.join(question.split()) 
            question = question.replace('&', ' ') 
            question = question.replace('\'', ' ') 
            question = question.replace(',', ';') 
 
            answer = answers[id - 1] 
 
            answer = re.sub(r'<[^>]*?>', '', str(answer.encode('utf8'))) 
            answer = ' '.join(answer.split()) 
            answer = answer.replace('&', ' ') 
            answer = answer.replace('\'', ' ') 
            answer = answer.replace(',', ';') 
 
            file = open('crawled.xml', 'a') 
 
            file.write(question) 
 
            file.write(",") 
            file.write(answer) 
 
            file.write("\n") 
 
            file.close() 
 
            id = id + 1 

我试过了

hxs.select('//INPUT/@onClick').extract() 

但是还是不行。路径有什么问题？

请注意，问题已成功提取。问题和答案的结构非常相似。为什么答案是空的？

请您参考如下方法:

首先，你获取答案的xpath表达式不正确，而不是

//td[3]/p/INPUT/@onClick 

使用

//td[3]/p/input/@onclick 

此外，这是我的蜘蛛版本:

from scrapy.spider import BaseSpider 
from scrapy.selector import HtmlXPathSelector 
from scrapy.item import Item, Field 
 
 
class DmozItem(Item): 
    number = Field() 
    question = Field() 
    answer = Field() 
 
 
class DmozSpider(BaseSpider): 
    name = "dmoz" 
    allowed_domains = ["2345.com"] 
    start_urls = ["http://www.2345.com/jzw/1.htm"] 
 
    def parse(self, response): 
        hxs = HtmlXPathSelector(response) 
 
        rows = hxs.select('//body/center/table[2]/tr') 
        for row in rows: 
            item = DmozItem() 
            try: 
                item['number'] = row.select(".//td[1]/p/font/text()").extract()[0] 
                item['question'] = row.select(".//td[2]/font/text()").extract()[0] 
                item['answer'] = row.select(".//td[3]/p/input/@onclick").extract()[0][13:-2] 
            except: 
                continue 
            yield item 

通过 scrapy runspider <spider_name.py> --output-format csv --output output.csv 运行它并在 output.csv 中查看 csv 格式的结果文件。

希望对您有所帮助。