我正在尝试使用 scrapy 编写我的第一个网络爬虫/数据提取器,但无法通过链接获取它。我也收到一个错误:
ERROR: Spider error processing < GET https://en.wikipedia.org/wiki/Wikipedia:Unusual_articles>
我知道蜘蛛正在扫描页面一次,因为我能够从我弄乱的 a 标签和 h1 元素中提取信息。
有谁知道我怎样才能使它跟随页面上的链接并消除错误?
import scrapy
from scrapy.linkextractors import LinkExtractor
from wikiCrawler.items import WikicrawlerItem
from scrapy.spiders import Rule
class WikispyderSpider(scrapy.Spider):
name = "wikiSpyder"
allowed_domains = ['https://en.wikipedia.org/']
start_urls = ['https://en.wikipedia.org/wiki/Wikipedia:Unusual_articles']
rules = (
Rule(LinkExtractor(canonicalize=True, unique=True), follow=True, callback="parse"),
)
def start_requests(self):
for url in self.start_urls:
yield scrapy.Request(url, callback=self.parse, dont_filter=True)
def parse(self, response):
items = []
links = LinkExtractor(canonicalize=True, unique=True).extract_links(response)
for link in links:
item = WikicrawlerItem()
item['url_from'] = response.url
item['url_to'] = link.url
items.append(item)
print(items)
return items
请您参考如下方法:
如果你想使用链接提取器,你需要使用一个特殊的蜘蛛类 - CrawlSpider :
from scrapy.spiders import CrawlSpider
class WikispyderSpider(CrawlSpider):
# ...
这是一个简单的蜘蛛程序,它跟踪您的起始 url 中的链接并打印出页面标题:
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider
from scrapy.spiders import Rule
class WikispyderSpider(CrawlSpider):
name = "wikiSpyder"
allowed_domains = ['en.wikipedia.org']
start_urls = ['https://en.wikipedia.org/wiki/Wikipedia:Unusual_articles']
rules = (
Rule(LinkExtractor(canonicalize=True, unique=True), follow=True, callback="parse_link"),
)
def parse_link(self, response):
print(response.xpath("//title/text()").extract_first())
