Skip to main content
 首页 » 编程设计

python Scrapy CrawlSpider : Rule doesn't apply after logged in, 只抓取第一页

2024年11月01日25jyk

我是一名 C/C++ 程序员,在制作绘图和文本处理方面的 Python 经验有限。我目前正在从事个人数据分析项目,我正在使用 Scrapy 抓取论坛中的所有线程和用户信息。

我整理了一个初始代码,旨在首先登录,然后从子论坛的索引页面开始,执行以下操作:

1) 提取所有包含“topic”的线程链接

2) 暂时将页面保存在一个文件中(一旦整个过程成功,将提取项目信息)

3) 找到标签class=next的下一页链接,转到下一页并重复1)和2)

我知道对于每个线程,我仍然需要浏览包含所有回复帖子的所有页面,但我计划在我当前的代码工作正常后进行此操作。

但是,我当前的代码只会提取起始 url 中的所有线程,然后停止。我已经搜索了几个小时,但没有找到任何解决方案。所以我在这里问我的问题,希望有 Scrapy 经验的人可以帮助我。如果你们想要任何其他信息,例如输出,请告诉我。谢谢!

关于 Paul 的回复,我更新了我的代码,我的链接提取器有问题,我需要修复它。除此之外,该规则现在运行良好。再次感谢Paul的帮助。

这是我当前的爬虫代码:

from scrapy.contrib.spiders import CrawlSpider 
from scrapy.http import Request, FormRequest 
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor 
from scrapy.contrib.spiders import Rule 
from scrapy.selector import Selector 
 
class ZhuaSpider(CrawlSpider): 
    name = 'zhuaspider' 
    allowed_domains = ['depressionforums.org'] 
    login_page = 'http://www.domain.com/forums/index.php?app=core&module=global&section=login' 
    start_urls = ['http://www.depressionforums.org/forums/forum/12-depression-central/'] 
 
    rules = (Rule(SgmlLinkExtractor(restrict_xpaths=('//li[@class="next"]'), unique=True), 
                           callback='parse_links', 
                           follow=True), 
            ) 
 
    def start_requests(self): 
        """called before crawling starts. Try to login""" 
        yield Request( 
                url=self.login_page, 
                callback=self.login, 
                dont_filter=True) 
 
    def login(self, response): 
        """Generate a login request.""" 
        return FormRequest.from_response(response, 
                formdata={'ips_username': 'myuid', 'ips_password': 'mypwd'}, 
                callback=self.check_login_response) 
 
    def check_login_response(self, response): 
        """Check the response returned by a login request to see if we are successfully logged in.""" 
        if "Username or password incorrect" in response.body: 
            self.log("Login failed.") 
        else: 
            self.log("Successfully logged in. Let's start crawling!") 
            # Now the crawling can begin. 
            for url in self.start_urls: 
                # explicitly ask Scrapy to run the responses through rules 
                yield Request(url, callback=self.parse) 
 
    def parse_links(self, response): 
        hxs = Selector(response) 
        links = hxs.xpath('//a[contains(@href, "topic")]') 
        for link in links: 
            title = ''.join(link.xpath('./@title').extract()) 
            url = ''.join(link.xpath('./@href').extract()) 
            meta={'title':title,} 
            yield Request(url, callback = self.parse_posts, meta=meta,) 
 
    #If I add this line it will only crawl the starting url, 
    #otherwise it still won't apply the rule and crawls nothing. 
    parse_start_url = parse_links 
 
    def parse_posts(self, response): 
        filename = 'download/'+ response.url.split("/")[-2] 
        open(filename, 'wb').write(response.body) 

请您参考如下方法:

利用CrawlSpider的 Rules , 你需要你的 Requests由内部处理 parse()方法。

您可以通过显式设置 callback=self.parse 来做到这一点或者不设置回调。

start_urls = ['http://www.depressionforums.org/forums/forum/12-depression-central/'] 
 
rules = ( 
    Rule(SgmlLinkExtractor(restrict_xpaths=('//li[@class="next"]'), unique=True), 
                           callback='parse_links', 
                           follow=True), 
) 
 
... 
 
def check_login_response(self, response): 
    """Check the response returned by a login request to see if we are successfully logged in.""" 
    if "Username or password incorrect" in response.body: 
        self.log("Login failed.") 
    else: 
        self.log("Successfully logged in. Let's start crawling!") 
        # Now the crawling can begin. 
        for url in self.start_urls: 
            # explicitly ask Scrapy to run the responses through rules 
            yield Request(url, callback=self.parse) 

然后,仅此一项,您应该会在 //li[@class="next"] 中看到链接页面使用 parse_links() 抓取和解析的部分...期待 start_urls 本身。

通过parse_links对于 start_urls,您必须定义一个特殊的 parse_start_url 属性。

你可以这样做:

def parse_links(self, response): 
    hxs = Selector(response) 
    links = hxs.xpath('//a[contains(@href, "topic")]') 
    for link in links: 
        title = ''.join(link.xpath('./@title').extract()) 
        url = ''.join(link.xpath('./@href').extract()) 
        meta={'title':title,} 
        yield Request(url, callback = self.parse_posts, meta=meta,) 
 
parse_start_url = parse_links