Skip to main content
 首页 » 编程设计

python之HTTP 状态代码未处理或不允许

2025年02月15日8qq78292959

我需要使用 Scrapy 抓取带有 cookie 的站点,但返回错误

代码在这里

class XueqiuSpider(scrapy.Spider): 
    name = "xueqiu" 
    start_urls = ["https://xueqiu.com/stock/f10/finmainindex.json?symbol=SZ000001&page=1&size=1"] 
    delimiter = ',' 
    quotechar = '"' 
    headers = ["symbol","date","open","high","low","close","volume"] 
 
    def start_requests(self): 
        for i,url in enumerate(self.start_urls): 
            print(url) 
            yield Request(url,cookies={'aliyungf_tc':'AQAAANiAQ3xQ/QAAZ0J2fRFnxcJufEzG'},callback=self.parse_item) 
 
    def parse_item(self, response): 
        print response 

错误显示

********Current UserAgent:Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11************ 
2017-03-02 18:56:02 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <GET https://xueqiu.com/stock/f10/finmainindex.json?symbol=SZ000001&page=1&size=1> 
Cookie: aliyungf_tc=AQAAANiAQ3xQ/QAAZ0J2fRFnxcJufEzG; aliyungf_tc=AQAAAM/c+1g5vAMAZ0J2fbusPyBy7jb1 
 
2017-03-02 18:56:02 [scrapy.core.engine] DEBUG: Crawled (400) <GET https://xueqiu.com/stock/f10/finmainindex.json?symbol=SZ000001&page=1&size=1> (referer: None) 
2017-03-02 18:56:02 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <400 https://xueqiu.com/stock/f10/finmainindex.json?symbol=SZ000001&page=1&size=1>: HTTP status code is not handled or not allowed 

请您参考如下方法:

正如您在 doc 中所读到的那样,根据HTTP标准,成功的响应是那些状态码在200-300范围内的响应。

如果您仍想处理超出该范围的响应代码,您可以使用 handle_httpstatus_list 爬虫属性或 HTTPERROR_ALLOWED_CODES 设置指定爬虫能够处理的响应代码。

所以你应该在你的代码中添加它

class XueqiuSpider(scrapy.Spider): 
    handle_httpstatus_list = [400]