Skip to main content
 首页 » 编程设计

python之Selenium/BeautifulSoup之Python之循环遍历多个页面

2024年12月31日7duanxz

我一天中大部分时间都在研究和测试在零售商网站上循环浏览一组产品的最佳方式。

虽然我能够成功地在第一页上收集到一组产品(和属性),但我一直难以找到循环遍历网站页面以继续我的抓取的最佳方法。

根据下面的代码,我尝试使用“while”循环和 Selenium 单击网站的“下一页”按钮,然后继续收集产品。

问题是我的代码仍然没有通过第 1 页。

我在这里犯了一个愚蠢的错误吗?阅读本网站上的 4 或 5 个类似示例,但没有一个足够具体以在此处提供解决方案。

from selenium import webdriver 
from bs4 import BeautifulSoup 
driver = webdriver.Chrome() 
driver.get('https://www.kohls.com/catalog/mens-button-down-shirts-tops-clothing.jsp?CN=Gender:Mens+Silhouette:Button-Down%20Shirts+Category:Tops+Department:Clothing&cc=mens-TN3.0-S-buttondownshirts&kls_sbp=43160314801019132980443403449632772558&PPP=120&WS=0') 
 
products.clear() 
hyperlinks.clear() 
reviewCounts.clear() 
starRatings.clear() 
 
products = [] 
hyperlinks = [] 
reviewCounts = [] 
starRatings = [] 
 
pageCounter = 0 
maxPageCount = int(html_soup.find('a', class_ = 'totalPageNum').text)+1 
 
 
html_soup = BeautifulSoup(driver.page_source, 'html.parser') 
prod_containers = html_soup.find_all('li', class_ = 'products_grid') 
 
 
while (pageCounter < maxPageCount): 
    for product in prod_containers: 
        # If the product has review count, then extract: 
        if product.find('span', class_ = 'prod_ratingCount') is not None: 
            # The product name 
            name = product.find('div', class_ = 'prod_nameBlock') 
            name = re.sub(r"\s+", " ", name.text) 
            products.append(name) 
 
            # The product hyperlink 
            hyperlink = product.find('span', class_ = 'prod_ratingCount') 
            hyperlink = hyperlink.a 
            hyperlink = hyperlink.get('href') 
            hyperlinks.append(hyperlink) 
 
            # The product review count 
            reviewCount = product.find('span', class_ = 'prod_ratingCount').a.text 
            reviewCounts.append(reviewCount) 
 
            # The product overall star ratings 
            starRating = product.find('span', class_ = 'prod_ratingCount') 
            starRating = starRating.a 
            starRating = starRating.get('alt') 
            starRatings.append(starRating)  
 
    driver.find_element_by_xpath('//*[@id="page-navigation-top"]/a[2]').click() 
    counterProduct +=1 
    print(counterProduct) 

请您参考如下方法:

每次“点击”下一页时都需要解析。因此,您需要将其包含在 while 循环中,否则您将继续遍历第一页,即使它点击到下一页也是如此,因为 prod_containers 对象永远不会改变。

其次,按照您的方式,您的 while 循环将永远不会停止,因为您设置了 pageCounter = 0,但永远不会递增它...它将永远是 <您的 maxpagecount。< p>

我修复了代码中的这两件事并运行了它,它似乎已经工作并解析了第 1 到 5 页。

from selenium import webdriver 
from bs4 import BeautifulSoup 
import re 
 
driver = webdriver.Chrome() 
driver.get('https://www.kohls.com/catalog/mens-button-down-shirts-tops-clothing.jsp?CN=Gender:Mens+Silhouette:Button-Down%20Shirts+Category:Tops+Department:Clothing&cc=mens-TN3.0-S-buttondownshirts&kls_sbp=43160314801019132980443403449632772558&PPP=120&WS=0') 
 
products = [] 
hyperlinks = [] 
reviewCounts = [] 
starRatings = [] 
 
pageCounter = 0 
 
html_soup = BeautifulSoup(driver.page_source, 'html.parser') 
maxPageCount = int(html_soup.find('a', class_ = 'totalPageNum').text)+1 
 
prod_containers = html_soup.find_all('li', class_ = 'products_grid') 
 
 
while (pageCounter < maxPageCount): 
    html_soup = BeautifulSoup(driver.page_source, 'html.parser') 
    prod_containers = html_soup.find_all('li', class_ = 'products_grid') 
    for product in prod_containers: 
        # If the product has review count, then extract: 
        if product.find('span', class_ = 'prod_ratingCount') is not None: 
            # The product name 
            name = product.find('div', class_ = 'prod_nameBlock') 
            name = re.sub(r"\s+", " ", name.text) 
            name = name.strip() 
            products.append(name) 
 
            # The product hyperlink 
            hyperlink = product.find('span', class_ = 'prod_ratingCount') 
            hyperlink = hyperlink.a 
            hyperlink = hyperlink.get('href') 
            hyperlinks.append(hyperlink) 
 
            # The product review count 
            reviewCount = product.find('span', class_ = 'prod_ratingCount').a.text 
            reviewCounts.append(reviewCount) 
 
            # The product overall star ratings 
            starRating = product.find('span', class_ = 'prod_ratingCount') 
            starRating = starRating.a 
            starRating = starRating.get('alt') 
            starRatings.append(starRating)  
 
    driver.find_element_by_xpath('//*[@id="page-navigation-top"]/a[2]').click() 
    pageCounter +=1 
    print(pageCounter)