Skip to main content
 首页 » 编程设计

python之Scrapy Authenticated Spider 获取内部服务器错误

2025年04月02日18del

我正在尝试制作经过身份验证的蜘蛛。我在这里提到了几乎所有与 Scrapy 身份验证蜘蛛相关的帖子,我找不到我的问题的任何答案。我使用了以下代码:

import scrapy 
 
from scrapy.spider import BaseSpider 
from scrapy.selector import  Selector 
from scrapy.http import FormRequest, Request 
import  logging 
from PWC.items import PwcItem 
 
 
class PwcmoneySpider(scrapy.Spider): 
    name = "PWCMoney" 
    allowed_domains = ["pwcmoneytree.com"] 
    start_urls = ( 
        'https://www.pwcmoneytree.com/SingleEntry/singleComp?compName=Addicaid', 
    ) 
 
    def parse(self, response): 
        return [scrapy.FormRequest("https://www.pwcmoneytree.com/Account/Login", 
                                   formdata={'UserName': 'user', 'Password': 'pswd'}, 
                                   callback=self.after_login)] 
 
    def after_login(self, response): 
      if "authentication failed" in response.body: 
        self.log("Login failed", level=logging.ERROR) 
        return 
    # We've successfully authenticated, let's have some fun! 
    print("Login Successful!!") 
    return Request(url="https://www.pwcmoneytree.com/SingleEntry/singleComp?compName=Addicaid", 
               callback=self.parse_tastypage) 
 
 
    def parse_tastypage(self, response): 
      for sel in response.xpath('//div[@id="MainDivParallel"]'): 
                                      item = PwcItem() 
                                      item['name'] = sel.xpath('div[@id="CompDiv"]/h2/text()').extract() 
                                      item['location'] = sel.xpath('div[@id="CompDiv"]/div[@id="infoPane"]/div[@class="infoSlot"]/div/a/text()').extract() 
                                      item['region'] = sel.xpath('div[@id="CompDiv"]/div[@id="infoPane"]/div[@id="contactInfoDiv"]/div[1]/a[2]/text()').extract() 
                                      yield item 

我得到了以下输出:

Microsoft Windows [Version 6.1.7601] 
Copyright (c) 2009 Microsoft Corporation.  All rights reserved. 
 
C:\Python27\PWC>scrapy crawl PWCMoney -o test.csv 
2016-04-29 11:37:35 [scrapy] INFO: Scrapy 1.0.5 started (bot: PWC) 
2016-04-29 11:37:35 [scrapy] INFO: Optional features available: ssl, http11 
2016-04-29 11:37:35 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'PW 
C.spiders', 'FEED_FORMAT': 'csv', 'SPIDER_MODULES': ['PWC.spiders'], 'FEED_URI': 
 'test.csv', 'BOT_NAME': 'PWC'} 
2016-04-29 11:37:35 [scrapy] INFO: Enabled extensions: CloseSpider, FeedExporter 
, TelnetConsole, LogStats, CoreStats, SpiderState 
2016-04-29 11:37:36 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddl 
eware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultH 
eadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMidd 
leware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats 
2016-04-29 11:37:36 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddlewa 
re, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware 
2016-04-29 11:37:36 [scrapy] INFO: Enabled item pipelines: 
2016-04-29 11:37:36 [scrapy] INFO: Spider opened 
2016-04-29 11:37:36 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 i 
tems (at 0 items/min) 
2016-04-29 11:37:36 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023 
2016-04-29 11:37:37 [scrapy] DEBUG: Retrying <POST https://www.pwcmoneytree.com/ 
Account/Login> (failed 1 times): 500 Internal Server Error 
2016-04-29 11:37:38 [scrapy] DEBUG: Retrying <POST https://www.pwcmoneytree.com/ 
Account/Login> (failed 2 times): 500 Internal Server Error 
2016-04-29 11:37:38 [scrapy] DEBUG: Gave up retrying <POST https://www.pwcmoneyt 
ree.com/Account/Login> (failed 3 times): 500 Internal Server Error 
2016-04-29 11:37:38 [scrapy] DEBUG: Crawled (500) <POST https://www.pwcmoneytree 
.com/Account/Login> (referer: None) 
2016-04-29 11:37:38 [scrapy] DEBUG: Ignoring response <500 https://www.pwcmoneyt 
ree.com/Account/Login>: HTTP status code is not handled or not allowed 
2016-04-29 11:37:38 [scrapy] INFO: Closing spider (finished) 
2016-04-29 11:37:38 [scrapy] INFO: Dumping Scrapy stats: 
{'downloader/request_bytes': 954, 
 'downloader/request_count': 3, 
 'downloader/request_method_count/POST': 3, 
 'downloader/response_bytes': 30177, 
 'downloader/response_count': 3, 
 'downloader/response_status_count/500': 3, 
 'finish_reason': 'finished', 
 'finish_time': datetime.datetime(2016, 4, 29, 6, 7, 38, 674000), 
 'log_count/DEBUG': 6, 
 'log_count/INFO': 7, 
 'response_received_count': 1, 
 'scheduler/dequeued': 3, 
 'scheduler/dequeued/memory': 3, 
 'scheduler/enqueued': 3, 
 'scheduler/enqueued/memory': 3, 
 'start_time': datetime.datetime(2016, 4, 29, 6, 7, 36, 193000)} 
2016-04-29 11:37:38 [scrapy] INFO: Spider closed (finished) 

由于我是 python 和 Scrapy 的新手,我似乎无法理解错误,希望这里的人能帮助我。

所以,我修改了这样的代码 Rejected的建议,只显示修改的部分:

allowed_domains = ["pwcmoneytree.com"] 
    start_urls = ( 
        'https://www.pwcmoneytree.com/Account/Login', 
    ) 
 
    def start_requests(self): 
        return [scrapy.FormRequest.from_response("https://www.pwcmoneytree.com/Account/Login", 
                                   formdata={'UserName': 'user', 'Password': 'pswd'}, 
                                   callback=self.logged_in)] 

出现如下错误:

C:\Python27\PWC>scrapy crawl PWCMoney -o test.csv 
2016-04-30 11:04:47 [scrapy] INFO: Scrapy 1.0.5 started (bot: PWC) 
2016-04-30 11:04:47 [scrapy] INFO: Optional features available: ssl, http11 
2016-04-30 11:04:47 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'PW 
C.spiders', 'FEED_FORMAT': 'csv', 'SPIDER_MODULES': ['PWC.spiders'], 'FEED_URI': 
 'test.csv', 'BOT_NAME': 'PWC'} 
2016-04-30 11:04:50 [scrapy] INFO: Enabled extensions: CloseSpider, FeedExporter 
, TelnetConsole, LogStats, CoreStats, SpiderState 
2016-04-30 11:04:54 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddl 
eware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultH 
eadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMidd 
leware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats 
2016-04-30 11:04:54 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddlewa 
re, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware 
2016-04-30 11:04:54 [scrapy] INFO: Enabled item pipelines: 
Unhandled error in Deferred: 
2016-04-30 11:04:54 [twisted] CRITICAL: Unhandled error in Deferred: 
 
 
Traceback (most recent call last): 
  File "c:\python27\lib\site-packages\scrapy\cmdline.py", line 150, in _run_comm 
and 
    cmd.run(args, opts) 
  File "c:\python27\lib\site-packages\scrapy\commands\crawl.py", line 57, in run 
 
    self.crawler_process.crawl(spname, **opts.spargs) 
  File "c:\python27\lib\site-packages\scrapy\crawler.py", line 153, in crawl 
    d = crawler.crawl(*args, **kwargs) 
  File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1274, in 
unwindGenerator 
    return _inlineCallbacks(None, gen, Deferred()) 
--- <exception caught here> --- 
  File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1128, in 
_inlineCallbacks 
    result = g.send(result) 
  File "c:\python27\lib\site-packages\scrapy\crawler.py", line 72, in crawl 
    start_requests = iter(self.spider.start_requests()) 
  File "C:\Python27\PWC\PWC\spiders\PWCMoney.py", line 16, in start_requests 
    callback=self.logged_in)] 
  File "c:\python27\lib\site-packages\scrapy\http\request\form.py", line 36, in 
from_response 
    kwargs.setdefault('encoding', response.encoding) 
exceptions.AttributeError: 'str' object has no attribute 'encoding' 
2016-04-30 11:04:54 [twisted] CRITICAL: 

请您参考如下方法:

从您的错误日志中可以看出,是对 https://www.pwcmoneytree.com/Account/Login 的 POST 请求给您带来了 500 错误。

我尝试使用 POSTman 手动发出相同的 POST 请求.它给出了 500 错误代码和包含此错误消息的 HTML 页面:

The required anti-forgery cookie "__RequestVerificationToken" is not present.

这是许多 API 和网站用来防止 CSRF 攻击的功能。如果您仍想抓取网站,则必须先访问登录表单并在登录前获取正确的 cookie。