Skip to main content
 首页 » 编程设计

python之BeautifulSoup 和请求不会使用 .findAll() 产生预期结果

2025年02月15日14tuyile006

我一直在编写一段代码,用于从 Steam 市场(针对游戏 Unturned)检索元素列表及其相应价格。我正在使用 BeautifulSoup (bs4) 和请求库。到目前为止,这是我的代码:

for page_num in range(1,10): 
website = 'http://steamcommunity.com/market/search?appid=304930#p'+str(page_num)+'_popular_desc' 
r = requests.get(website) 
doc = r.text.split('\n') 
soup = BeautifulSoup(''.join(doc), "html.parser") 
 
names = soup.findAll("span", { "class" : "market_listing_item_name" }) 
for item in range(len(names)): 
    items.append(names[item].contents[0]) 
 
costs = soup.findAll("span", { "class" : "normal_price" }) 
for cost in range(len(costs)): 
    prices.append(costs[cost].contents[0]) 

预期输出:

Festive Gift Present :  $0.32 USD 
Halloween Gift Present :  $0.26 USD 
Carbon Fiber Mystery Box :  $0.47 USD 
Festive Hat :  $1.67 USD 
Nuclear Matamorez :  $0.39 USD 
... and so on 

此代码的问题在于,它仅获取第一页的名称。如果我用不同的数字代替 page_num 手动键入 URL,它会更改页面,HTML 文档也会更改。但是,代码似乎并没有从第二页等获取结果。 requests 每次都获取正确的 URL,但 HTML 文档返回相同的 URL?

请您参考如下方法:

第 2、3 等页面是通过 ajax(或类似的)请求的,因此当您首次加载页面时源代码不存在。为了绕过这个,我们可以嗅探 ajax url 并直接解析源代码,在这种情况下,json 编码,即:

import json 
from bs4 import BeautifulSoup 
from urllib2 import urlopen 
output = "" 
items =[] 
prices =[] 
for page_num in range(0,100, 10): # 
    start = page_num 
    count = page_num + 10 
 
    url = urlopen("http://steamcommunity.com/market/search/render/?query=&start={}&count={}&search_descriptions=0&sort_column=popular&sort_dir=desc&appid=304930".format(start, count)) 
    jsonCode = json.loads(url.read()) 
    output += jsonCode['results_html'] 
 
soup = BeautifulSoup(output, "html.parser") 
 
names = soup.findAll("span", { "class" : "market_listing_item_name" }) 
for item in range(len(names)): 
    items.append(names[item].contents[0]) 
 
costs = soup.findAll("span", { "class" : "normal_price" }) 
for cost in range(len(costs)): 
    if "Starting at" not in costs[cost].contents[0]: # we just get the first price 
        prices.append(costs[cost].contents[0]) 
 
 
 
print items 
[u'Festive Gift Present', u'Halloween Gift Present', u'Hypertech Timberwolf', u'Holiday Scarf', u'Chill Honeybadger', etc...]  
print prices 
[u'$0.34 USD', u'$0.28 USD', u'$1.77 USD', u'$0.31 USD', u'$0.65 USD', etc...] 

PS:Steam 将在约 50 次请求后临时禁止您的 IP