我目前正在使用 BS4 从 Kickstarter 网页提取一些信息:https://www.kickstarter.com/projects/louisalberry/louis-alberry-debut-album-uk-european-tour
项目信息位于其中一个脚本标签内:(伪代码)
...
<script>...</script>
<script>
window.current_ip = ...
...
window.current_project = "<I want this part>"
</script>
...
我当前的代码:
from bs4 import BeautifulSoup
from urllib.request import urlopen
import html
html_ = urlopen("https://www.kickstarter.com/projects/louisalberry/louis-alberry-debut-album-uk-european-tour").read()
soup = BeautifulSoup(html_, 'html.parser')
# why does this not work?
# soup.find('script', re.compile("window.current_project"))
# currently, I'm doing this:
all_string = html.unescape(soup.find_all('script')[4].get_text())
# then some regex here on all_string to extract the current_project information
目前,我可以使用索引 [4] 获取我想要的部分,但由于我不确定这在一般情况下是否正确,我如何从正确的脚本标签中提取文本?
谢谢!
请您参考如下方法:
您可以收集所有脚本元素并循环。用请求访问响应对象内容
from bs4 import BeautifulSoup
import requests
res = requests.get("https://www.kickstarter.com/projects/louisalberry/louis-alberry-debut-album-uk-european-tour")
soup = BeautifulSoup(res.content, 'lxml')
scripts = soup.select('script')
scripts = [script for script in scripts]
for script in scripts:
if 'window.current_project' in script.text:
print(script)
