我正在尝试使用 Scrapy 从网站获取每个链接(而不是其他数据)。我想通过从主页开始,从那里抓取所有链接,然后对于找到的每个链接,跟踪链接并从该页面抓取所有(唯一)链接,并对找到的所有链接执行此操作,直到没有更多跟随。
我还必须输入用户名和密码才能进入站点上的每个页面,因此我在 start_requests 中包含了一个基本的身份验证组件。
到目前为止,我有一个蜘蛛只给我主页上的链接,但我似乎无法弄清楚为什么它不跟踪链接并抓取其他页面。
这是我的蜘蛛:
from examplesite.items import ExamplesiteItem
import scrapy
from scrapy.linkextractor import LinkExtractor
from scrapy.spiders import Rule, CrawlSpider
from scrapy import Request
from w3lib.http import basic_auth_header
from scrapy.crawler import CrawlerProcess
class ExampleSpider(CrawlSpider):
#name of crawler
name = "examplesite"
#only scrape on pages within the example.co.uk domain
allowed_domains = ["example.co.uk"]
#start scraping on the site homepage once credentials have been authenticated
def start_requests(self):
url = str("https://example.co.uk")
username = "*********"
password = "*********"
auth = basic_auth_header(username, password)
yield scrapy.Request(url=url,headers={'Authorization': auth})
#rules for recursively scraping the URLS found
rules = [
Rule(
LinkExtractor(
canonicalize=True,
unique=True
),
follow=True,
callback="parse"
)
]
#method to identify hyperlinks by xpath and extract hyperlinks as scrapy items
def parse(self, response):
for element in response.xpath('//a'):
item = ExamplesiteItem()
oglink = element.xpath('@href').extract()
#need to add on prefix as some hrefs are not full https URLs and thus cannot be followed for scraping
if "http" not in str(oglink):
item['link'] = "https://example.co.uk" + oglink[0]
else:
item['link'] = oglink
yield item
这是我的项目类:
from scrapy import Field, Item
class ExamplesiteItem(Item):
link = Field()
我认为我出错的地方是“规则”,我知道你需要点击链接,但我不完全理解它是如何工作的(已经尝试在线阅读一些解释但仍然没有当然)。
如有任何帮助,我们将不胜感激!
请您参考如下方法:
您的规则没问题,问题是覆盖了 parse 方法。
来自 https://doc.scrapy.org/en/latest/topics/spiders.html#crawling-rules 的 scrapy 文档
When writing crawl spider rules, avoid using
parseas callback, since theCrawlSpideruses theparsemethod itself to implement its logic. So if you override theparsemethod, the crawl spider will no longer work.
