Skip to main content
 首页 » 编程设计

python 将 beautifulsoup 输出转换为 dict/json

2024年12月31日8leader

我正在尝试从亚马逊的产品页面抓取数据。我用 beautifulsoup 得到了整个标记。我想以以下json格式获取必要的产品详细信息

{ 
  asin: string, 
  title: string, 
  price: number, 
  listPrice: number, 
  prime: boolean, 
 
  dimensions: { 
    height: number, 
    length: number, 
    width: number, 
    weight: number, 
  }, 
  images: Array<string>, 
  attributes: Array<{ name: string, value: string }>, 
  categories: <{ node: string, title: string }>, 
 
} 

据我了解,我需要先获取字典格式的详细信息。但不确定如何从巨大的 html 中获取这些特定标签以将它们转换为字典。

编辑:我的代码看起来像这样

import requests 
from bs4 import BeautifulSoup 
 
url = "http://www.amazon.com/dp/B00ILZH9BO" 
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'} 
response = requests.get(url,headers=headers) 
soup = BeautifulSoup(response.text,"lxml") 
print(soup) 

编辑 2:我提供了一些我需要产品详细信息的 html

#######title######### 
<span class="a-size-large" id="productTitle"> 
  MagicBrite Complete Teeth Whitening Kit At Home Whitening 
</span> 
 
 
  #########price##### 
<span class="a-color-price"> 
    <span class="p13n-sc-price">$11.85</span> 
</span> 
 
 
 
  ############images######### 
 
<li class="a-spacing-small item"><span class="a-list-item"> 
<span class="a-declarative" data-action="thumb-action" data-thumb-action='{"thumbnailIndex":4,"variant":"PT04","index":4,"type":"image"}'> 
<span class="a-button a-button-thumbnail a-button-toggle"><span class="a-button-inner"><input class="a-button-input" type="submit"/><span aria-hidden="true" class="a-button-text"> 
<img alt="" src="https://images-na.ssl-images-amazon.com/images/I/51f8kCdwmqL._SS40_.jpg"/> 
</span></span></span> 
</span> 
</span></li> 
<li class="a-spacing-small item"><span class="a-list-item"> 
<span class="a-declarative" data-action="thumb-action" data-thumb-action='{"thumbnailIndex":5,"variant":"PT05","index":5,"type":"image"}'> 
<span class="a-button a-button-thumbnail a-button-toggle"><span class="a-button-inner"><input class="a-button-input" type="submit"/><span aria-hidden="true" class="a-button-text"> 
<img alt="" src="https://images-na.ssl-images-amazon.com/images/I/517mTOTBEiL._SS40_.jpg"/> 
</span></span></span> 
</span> 
</span></li> 

请您参考如下方法:

手动。

data = { 
    'asin': soup.find(id="ASIN").attrs['value'], 
    'title': soup.find(id="title").text.strip(), 
    'price': soup.find(id="price").find(id="priceblock_ourprice").text.strip(), 
    .... 
} 

price 似乎有点隐藏,在不同的页面上可能会有所不同,在哪里可以找到实际的“最终价格”。

最后,一旦您准备好 dict,只需将其提供给 json.dumps()。

import json 
result = json.dumps(data) 

如果亚马逊决定改变他们的加价,事情可能会破裂。