我正在使用 DOWNLOADER_MIDDLEWARES 通过 scrapy.Spider 旋转代理,我想得到一个项目,即 item['proxy_used'] ,用于每个请求的代理。
我想可以通过“统计收集器”获取代理,但我是 Python 和 Scrapy 的新手,直到现在我还没有找到解决方案。
import scrapy
from tutorial.items import QuotesItem
class QuotesSpider(scrapy.Spider):
name = "quotes"
allowed_domains = ["quotes.toscrape.com"]
start_urls = [
'http://quotes.toscrape.com/',
]
def parse_quotes(self, response):
for sel in response.css('div.quote'):
item = QuotesItem()
item['text'] = sel.css('span.text::text').get()
item['author'] = sel.css('small.author::text').get()
item['tags'] = sel.css('div.tags a.tag::text').getall()
item['quotelink'] = sel.css('small.author ~ a[href*="goodreads.com"]::attr(href)').get()
item['proxy_used'] = ??? <-- PROXY USED BY REQUEST - "HOW TO???"
yield item
# follow pagination links @shortcut
for a in response.css('li.next a'):
yield response.follow(a, callback = self.parse_quotes)
请您参考如下方法:
您可以使用响应对象来访问所使用的代理。如下图
response.meta.get("proxy")
也在您的代码中进行了更新。
import scrapy
from tutorial.items import QuotesItem
class QuotesSpider(scrapy.Spider):
name = "quotes"
allowed_domains = ["quotes.toscrape.com"]
start_urls = [
'http://quotes.toscrape.com/',
]
def parse_quotes(self, response):
for sel in response.css('div.quote'):
item = QuotesItem()
item['text'] = sel.css('span.text::text').get()
item['author'] = sel.css('small.author::text').get()
item['tags'] = sel.css('div.tags a.tag::text').getall()
item['quotelink'] = sel.css('small.author ~ a[href*="goodreads.com"]::attr(href)').get()
item['proxy_used'] = response.meta.get("proxy")
yield item
# follow pagination links @shortcut
for a in response.css('li.next a'):
yield response.follow(a, callback = self.parse_quotes)
