我一直在编写一段代码,用于从 Steam 市场(针对游戏 Unturned)检索元素列表及其相应价格。我正在使用 BeautifulSoup (bs4) 和请求库。到目前为止,这是我的代码:
for page_num in range(1,10):
website = 'http://steamcommunity.com/market/search?appid=304930#p'+str(page_num)+'_popular_desc'
r = requests.get(website)
doc = r.text.split('\n')
soup = BeautifulSoup(''.join(doc), "html.parser")
names = soup.findAll("span", { "class" : "market_listing_item_name" })
for item in range(len(names)):
items.append(names[item].contents[0])
costs = soup.findAll("span", { "class" : "normal_price" })
for cost in range(len(costs)):
prices.append(costs[cost].contents[0])
预期输出:
Festive Gift Present : $0.32 USD
Halloween Gift Present : $0.26 USD
Carbon Fiber Mystery Box : $0.47 USD
Festive Hat : $1.67 USD
Nuclear Matamorez : $0.39 USD
... and so on
此代码的问题在于,它仅获取第一页的名称。如果我用不同的数字代替 page_num 手动键入 URL,它会更改页面,HTML 文档也会更改。但是,代码似乎并没有从第二页等获取结果。 requests 每次都获取正确的 URL,但 HTML 文档返回相同的 URL?
请您参考如下方法:
第 2、3 等页面是通过 ajax(或类似的)请求的,因此当您首次加载页面时源代码不存在。为了绕过这个,我们可以嗅探 ajax url 并直接解析源代码,在这种情况下,json 编码,即:
import json
from bs4 import BeautifulSoup
from urllib2 import urlopen
output = ""
items =[]
prices =[]
for page_num in range(0,100, 10): #
start = page_num
count = page_num + 10
url = urlopen("http://steamcommunity.com/market/search/render/?query=&start={}&count={}&search_descriptions=0&sort_column=popular&sort_dir=desc&appid=304930".format(start, count))
jsonCode = json.loads(url.read())
output += jsonCode['results_html']
soup = BeautifulSoup(output, "html.parser")
names = soup.findAll("span", { "class" : "market_listing_item_name" })
for item in range(len(names)):
items.append(names[item].contents[0])
costs = soup.findAll("span", { "class" : "normal_price" })
for cost in range(len(costs)):
if "Starting at" not in costs[cost].contents[0]: # we just get the first price
prices.append(costs[cost].contents[0])
print items
[u'Festive Gift Present', u'Halloween Gift Present', u'Hypertech Timberwolf', u'Holiday Scarf', u'Chill Honeybadger', etc...]
print prices
[u'$0.34 USD', u'$0.28 USD', u'$1.77 USD', u'$0.31 USD', u'$0.65 USD', etc...]
PS:Steam 将在约 50 次请求后临时禁止您的 IP
