Skip to main content
 首页 » 编程设计

python之在 scrapy 中从 POST 请求下载文件

2024年11月24日12langtianya

我知道有内置的中间件来处理下载。但它只接受一个网址。但就我而言,我的下载链接是一个 POST 请求。

当我发出 POST 请求时,pdf 文件开始下载。

现在我想在 scrapy 中通过 POST 请求下载该文件。

网站是http://scrb.bihar.gov.in/View_FIR.aspx 您可以进入区 Aurangabad 和警察局 Kasma PS

在最后一列 status 上有一个下载文件的链接。

        ps_x = '//*[@id="ctl00_ContentPlaceHolder1_ddlPoliceStation"]//option[.="Kasma PS"]/@value' 
        police_station_val = response.xpath(ps_x).extract_first() 
        d_x = '//*[@id="ctl00_ContentPlaceHolder1_ddlDistrict"]//option[.="Aurangabad"]/@value' 
        district_val = response.xpath(d_x).extract_first() 
        viewstate = response.xpath(self.viewstate_x).extract_first() 
        viewstategen = response.xpath(self.viewstategen_x).extract_first() 
        eventvalidator = response.xpath(self.eventvalidator_x).extract_first() 
        eventtarget = response.xpath(self.eventtarget_x).extract_first() 
        eventargs = response.xpath(self.eventargs_x).extract_first() 
        lastfocus = response.xpath(self.lastfocus_x).extract_first() 
        payload = { 
            '__EVENTTARGET': eventtarget, 
            '__EVENTARGUMENT': eventargs, 
            '__LASTFOCUS': lastfocus, 
            '__VIEWSTATE': viewstate, 
            '__VIEWSTATEGENERATOR': viewstategen, 
            '__EVENTVALIDATION': eventvalidator, 
            'ctl00$ContentPlaceHolder1$ddlDistrict': district_val, 
            'ctl00$ContentPlaceHolder1$ddlPoliceStation': police_station_val, 
            'ctl00$ContentPlaceHolder1$optionsRadios': 'radioPetioner', 
            'ctl00$ContentPlaceHolder1$txtSearchBy': '', 
            'ctl00$ContentPlaceHolder1$rptItem$ctl06$lnkStatus.x': '21', 
            'ctl00$ContentPlaceHolder1$rptItem$ctl06$lnkStatus.y': '24', 
        } 
        headers = { 
            'Connection': 'keep-alive', 
            'Cache-Control': 'max-age=0', 
            'Origin': 'http://scrb.bihar.gov.in', 
            'Upgrade-Insecure-Requests': '1', 
            'Content-Type': 'application/x-www-form-urlencoded', 
            'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36', 
            'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8', 
            'Referer': 'http://scrb.bihar.gov.in/View_FIR.aspx', 
            'Accept-Encoding': 'gzip, deflate', 
            'Accept-Language': 'en-US,en;q=0.9', 
        } 
        # req = requests.post(response.url, data=payload, headers=headers) 
        # with open('pdf/ch.pdf', 'w+b') as f: 
        #     f.write(req.content) 

请您参考如下方法:

当您单击下载时,网络浏览器会发送 POST 请求。 所以这个 answer mentioned by El Ruso earlier适用于您的情况

..... 
    def parse(self, response): 
    ...... 
        yield scrapy.FormRequest("http://scrb.bihar.gov.in/View_FIR.aspx",.#your post request configuration, callback=self.save_pdf) 
 
    def save_pdf(self, response): 
        path = response.url.split('/')[-1] 
        self.logger.info('Saving PDF %s', path) 
        with open(path, 'wb') as f: 
            f.write(response.body)