Skip to main content
 首页 » 编程设计

python之使用 Beautfiul Soup 提取当前出价金额之动态更新值

2024年12月31日10bjzhanghao

这是我对 Python 和 BeautifulSoup 的第一次介绍。我试图从一个流行的拍卖网站 (RealInsight) 上列出的特定属性中获取当前的出价金额,但我无法让 BeautifulSoup 提取我正在寻找的实际整数,只有 HTML 代码。我正在寻找“s-b-n”类标签的值(value),在拍卖真正开始之前,它的值(value)是 3,250,000 美元。

https://marketplace.realinsight.com/sales/details/XXX

我认为这是因为该值是动态更新的并且是在 HTML 代码之外生成的,但我不确定如何验证该论点或如果证明正确则如何获取该值。我还认为我可能没有正确引用包含值的表,但同样,我在 python 或 bs4 方面经验不足。

[使用 ewwink 的方法使用 ewwink 的方法对下面的最终代码进行了更新 - 一次抓取一次,持续五秒] - 已更新以处理拍卖结束 -

import bs4 
import time 
import csv 
import datetime 
import sys 
from bs4 import BeautifulSoup as soup 
from urllib.request import urlopen as uReq 
my_url = 'https://marketplace.realinsight.com/sales/details/XXX' 
uclient = uReq(my_url) 
page_html = uclient.read() 
uclient.close() 
endmsg = "Auction End" 
page_soup = soup(page_html, "html.parser") 
propname = page_soup.title.text 
bids = page_soup.select_one(".body-content") 
currentbid = bids['data-nb'] 
bidincrement = bids['data-bi'] 
with open(propname + '_bids.csv','w',newline='') as f: 
    thewriter = csv.writer(f) 
    thewriter.writerow(['i','prop_name','date_time','bid_increment','bid_amt']) 
    for i in range(0,5,1): 
            try: 
                import sys 
                sale = page_soup.select_one("div.sale-end-text") 
                auctionend = sale.text.replace(" ", "") 
                if auctionend == sale.text.replace(" ", ""): 
                    currentDT = datetime.datetime.now()     
                    thewriter.writerow([i,endmsg,currentDT,currentbid])     
                    print(endmsg,currentbid) 
                    time.sleep(1) 
                    sys.exit() 
                else: 
                    print('will never get to this point') 
            except Exception: 
                pass 
 
            currentDT = datetime.datetime.now() 
            thewriter.writerow([i,propname,currentDT,bidincrement,currentbid]) 
            print(i,propname,currentDT,bidincrement,currentbid) 
            time.sleep(1) 

更新了 chitown88 的方法

import bs4 
import datetime 
import time 
import csv 
import selenium 
from selenium import webdriver 
driver = webdriver.Chrome(executable_path='C:\\Users\\XXXX\\Downloads\\chromedriver_win32\\chromedriver.exe') 
driver.get('https://marketplace.realinsight.com/sales/details/XXX') 
html = driver.page_source 
page_soup = bs4.BeautifulSoup(html,"html.parser") 
bids = page_soup.select("td.s-b-n") 
propname = page_soup.title.text 
currentbid = bids[0].text 
with open(propname + '_bids.csv','w',newline='') as f: 
    thewriter = csv.writer(f) 
    thewriter.writerow(['i','prop_name','date_time','bid_amt']) 
    for i in range(0, 5, 1): 
        currentDT = datetime.datetime.now() 
        driver.refresh() 
        thewriter.writerow([i, propname, currentDT, currentbid]) 
        print(i, propname, currentDT, currentbid) 
        time.sleep(1) 
driver.close() 

我可以在 HTML 代码中看到我要查找的数字 ($3,250,000),但它每隔几秒就会闪烁和更新一次,这就是为什么我认为它是在其他地方生成的。

任何指导将不胜感激。

请您参考如下方法:

您需要在解析之前加载页面。 Selenium非常适合。

import bs4  
from selenium import webdriver  
 
driver = webdriver.Chrome() 
driver.get('https://marketplace.realinsight.com/sales/details/367') 
 
html = driver.page_source 
page_soup = bs4.BeautifulSoup(html,"html.parser") 
 
bids = page_soup.select("td.s-b-n") 
bid = bids[0].text 
print(bid) 
 
driver.close() 

和输出:

In [91]: print(bid) 
$3,250,000