Skip to main content
 首页 » 编程设计

python scrapy 图像管道未下载(301 错误)

2025年01月19日8mate10pro

我正在尝试从本网站上的此类页面下载图像:http://39.moscowfilmfestival.ru/miff39/eng/films/?id=39016 .但我收到 301 错误,图像未下载。我可以毫无问题地下载所有其他数据点,包括 images_url。 (我正在重用在其他类似网站上使用过的 scrapy 代码。)如果我将下载的 images_url 输入到浏览器中,它会返回一个包含图像的页面。但是,页面的 URL 略有不同,插入了一个正斜杠 (/):

submit: http://39.moscowfilmfestival.ru/upimg/cache/photo/640/6521.jpg 
receive: http://moscowfilmfestival.ru/upimg//cache/photo/640/6521.jpg 

以上页面的输出日志如下:

2018-01-02 11:19:40 [selenium.webdriver.remote.remote_connection] DEBUG: POST http://127.0.0.1:62638/session/949ab9c1-6a0a-6a42-a19a-ef72c55acc33/url {"sessionId": "949ab9c1-6a0a-6a42-a19a-ef72c55acc33", "url": "http://39.moscowfilmfestival.ru//miff39/eng/films/?id=39016"} 
 
2018-01-02 14:46:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://39.moscowfilmfestival.ru//miff39/eng/films/?id=39016> (referer: None) 
 
2018-01-02 14:46:59 [scrapy.core.engine] DEBUG: Crawled (301) <GET http://39.moscowfilmfestival.ru/upimg/cache/photo/640/6521.jpg> (referer: None) 
2018-01-02 14:46:59 [scrapy.pipelines.files] WARNING: File (code: 301): Error downloading file from <GET http://39.moscowfilmfestival.ru/upimg/cache/photo/640/6521.jpg> referred in <None> 
2018-01-02 14:46:59 [scrapy.core.scraper] DEBUG: Scraped from <200 http://39.moscowfilmfestival.ru//miff39/eng/films/?id=39016> 
{'camera': ['HUANG LIAN'], 
 'cast': ['GAO ZIFENG, MENG HALYAN, JHAO ZIFENG, HE MIAO, WAN PEILU'], 
 'country': ['CHINA'], 
 'design': ['YANG ZHIWEN'], 
 'director': ['Liang Qiao'], 
 'festival_edition': ['39th'], 
 'festival_year': ['2017'], 
 'image_urls': ['http://39.moscowfilmfestival.ru/upimg/cache/photo/640/6521.jpg'], 
 'images': [], 
 'length': ['107'], 
 'music': [''], 
 'producer': ['DUAN PENG'], 
 'production': ['SUNNYWAY FILM'], 
 'program': ['Main Competition'], 
 'script': ['LI YONG'], 
 'sound': ['HU MAI, HAO CONG'], 
 'synopsis': ['The story begins with Vince Kang, a reporter in Beijing, having ' 
              'to go back to his hometown to report a crested ibis, one of the ' 
              'national treasures found unexpectedly. During the process of ' 
              'pursuit and hide of the crested ibis, everyone’s interest is ' 
              'revealed and the scars, both mental and physical were rip up. ' 
              'In addition, the environment pollution, an aftermath from ' 
              'China`s development pattern, is brought into daylight. The ' 
              'story, from the perspective of a returnee, reveals the living ' 
              'condition of rural China and exposes the dilemma of humanity. ' 
              'In the end, Vince, the renegade, had no alternative but make a ' 
              'compromise with his birthland.'], 
 'title': ['CRESTED IBIS'], 
 'year': ['2017']} 

解决问题:

  1. 我试图通过插入额外的 / 来模仿浏览器 url。没有影响。

  2. 我已尝试将 301 异常处理程序添加到蜘蛛类 (handle_httpstatus_all = True) 以及 settings.py 文件中。没有影响。

有趣的是,我写的蜘蛛的早期版本错误地用额外的 / 完成了部分 url(在 .ru 和 miff 之间) URL 的一部分),并且 GET 和 POST 请求工作正常。它们与当前版本的蜘蛛程序中正确的原始页面 URL 的工作方式相同。

真诚感谢任何帮助。

请您参考如下方法:

我建议你使用 urllib 库来下载任何图像。

import urllib 
from urllib import request 
url = 'http://39.moscowfilmfestival.ru/upimg/cache/photo/640/6521.jpg' 
file_path = r'C:/Users/admin/Desktop/test/6521.jpg' 
getPath, headers = urllib.request.urlretrieve(url, file_path) 
print(getPath) #This is the image path