Skip to main content
 首页 » 编程设计

python之Scrapy Xpath 构造在动态站点上生成空括号

2024年10月25日13cyq1162

我正在尝试通过 scrapy 创建一个蜘蛛来抓取网站并提取特定商店的所有链接。最终,蜘蛛将使用这些商店链接来提取价格信息。该站点旨在将商店信息分解为州和地区。我已经能够提取州/地区的所有子链接,但我似乎无法提取商店的特定信息,该信息是商店链接。

我认为这与页面加载的时间不够长,无法显示这些商店链接有关。所以我使用了 selenium 并将时间延迟了 20 秒,以便为网页加载留出适当的时间。一个示例链接是 https://weedmaps.com/dispensaries/in/united-states/colorado正如您在左侧看到的,有特定商店的信息。使用 inspect 元素的代码片段生成以下 HTML:

<li class="ng-scope" ng-repeat="listing in listings" wm-listing-detail="" role="listing"> 
  <div ng-class="{unpublished: !listing.published}"> 
     <div class="listing feature-1 recreational dispensary" ng-     class="featureClassesFor(listing)" ng-click="setActiveMarker(listing)"> 
         <img class="avatar" ng-src="https://d2kxqxnk1i5o9a.cloudfront.net/uploads/avatars/dispensaries/5566/square_10569095_10152344322971376_2924814837799131094_n.jpg" src="https://d2kxqxnk1i5o9a.cloudfront.net/uploads/avatars/dispensaries/5566/square_10569095_10152344322971376_2924814837799131094_n.jpg"/> 
         <div> 
           <div class="name hovers"> 
              <a class="ng-binding" href="/dispensaries/organic-alternatives" role="listing url" ng-bind-html="listing.name">Organic Alternatives</a> 
           </div> 
         </div> 
         <div class="address ng-binding"> 
         <div class="wm_map_inactive_rating"> 

我想提取@href 后面的信息。

我在 scrapy 项目的蜘蛛部分有以下 python 代码:

import scrapy 
from scrapy.spider import BaseSpider  
from selenium import webdriver 
from selenium.webdriver.support.ui import WebDriverWait 
 
class scrapybotspider(scrapy.Spider): 
    name = "scrapybot_spider" 
    start_urls = ['https://weedmaps.com/dispensaries/in/united-states/colorado'] 
    delay = 20 
 
    def __init__(self): 
        self.driver = webdriver.Firefox() 
 
    def parse(self, response): 
        self.driver.get(response.url) 
 
        while True: 
            StateCountyURL = response.xpath("//*[@id='wm-listings']/div/li[2]/div/div/div[1]/div/@href").extract() 
            print StateCountyURL 

这会产生空括号,我已经尝试了许多不同的 xpath 结构。

请您参考如下方法:

您需要开始使用 Explicit Waits .

等待列表可见。获取区域链接的工作示例代码:

from selenium import webdriver 
from selenium.webdriver.support.wait import WebDriverWait 
from selenium.webdriver.common.by import By 
from selenium.webdriver.support import expected_conditions as EC 
 
 
driver = webdriver.Firefox() 
driver.get('https://weedmaps.com/dispensaries/in/united-states/colorado') 
 
wait = WebDriverWait(driver, 10) 
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.subregion a.region-ajax-link"))) 
 
for region in driver.find_elements_by_css_selector("div.subregion a.region-ajax-link"): 
    print region.get_attribute("href") 
 
driver.close() 

打印:

https://weedmaps.com/dispensaries/in/aspen-colorado 
https://weedmaps.com/dispensaries/in/aurora 
https://weedmaps.com/dispensaries/in/boulder 
https://weedmaps.com/dispensaries/in/colorado-springs 
... 
https://weedmaps.com/dispensaries/in/other-co 
https://weedmaps.com/dispensaries/in/summit-eagle-county 
https://weedmaps.com/dispensaries/in/trinidad 

同样,对于药房:

wait = WebDriverWait(driver, 10) 
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.dispensary div.name a"))) 
 
for dispensary in driver.find_elements_by_css_selector("div.dispensary div.name a"): 
    print dispensary.get_attribute("href")