Skip to main content
 首页 » 编程设计

python之用 mechanicalsoup 下载文件

2024年12月31日7jiqing9006

我想下载这个ONS webpage上的Excel文件在 Python 中使用 MechanicalSoup 包。我读过 MechanicalSoup documentation .我在 StackOverflow 和其他地方广泛搜索了一个示例,但没有成功。

我的尝试是:

# Install dependencies 
# pip install requests 
# pip install BeautifulSoup4 
# pip install MechanicalSoup 
 
# Import libraries 
import mechanicalsoup 
import urllib.request 
import requests 
from bs4 import BeautifulSoup 
 
# Create a browser object that can collect cookies 
browser = mechanicalsoup.StatefulBrowser() 
 
browser.open("https://www.ons.gov.uk/economy/grossdomesticproductgdp/timeseries/l2kq/qna") 
 
browser.download_link("https://www.ons.gov.uk/generator?format=xls&uri=/economy/grossdomesticproductgdp/timeseries/l2kq/qna") 

在最后一行,我也尝试过:

browser.download_link(link="https://www.ons.gov.uk/generator?format=xls&uri=/economy/grossdomesticproductgdp/timeseries/l2kq/qna",file="c:/test/filename.xls") 

2019 年 1 月 25 日更新:感谢 AKX 在下方的评论,我已尝试过

browser.download_link(re.escape("https://www.ons.gov.uk/generator?format=xls&uri=/economy/grossdomesticproductgdp/timeseries/l2kq/qna")) 

在每种情况下,我都会收到错误:

mechanicalsoup.utils.LinkNotFoundError 

但链接确实存在。尝试将其粘贴到您的地址栏以确认:

https://www.ons.gov.uk/generator?format=xls&uri=/economy/grossdomesticproductgdp/timeseries/l2kq/qna 

我做错了什么?

2019 年 1 月 25 日更新 2:感谢 AKX 在下面的回答,这是回答我问题的完整 MWE(发布给以后遇到同样困难的人) :

# Install dependencies 
# pip install requests 
# pip install BeautifulSoup4 
# pip install MechanicalSoup 
 
# Import libraries 
import mechanicalsoup 
import urllib.request 
import requests 
from bs4 import BeautifulSoup 
import re 
 
# Create a browser object that can collect cookies 
browser = mechanicalsoup.StatefulBrowser() 
 
browser.open("https://www.ons.gov.uk/economy/grossdomesticproductgdp/timeseries/l2kq/qna") 
 
browser.download_link(link_text=".xls",file="c:/py/ONS_Data.xls" ) 

请您参考如下方法:

我没用过 Mechanical Soup,但是看了文档,

This function behaves similarly to follow_link()

和follow_link说(强调我的)

  • If link is a bs4.element.Tag (i.e. from a previous call to links() or find_link()), then follow the link.
  • If link doesn’t have a href-attribute or is None, treat link as a url_regex and look it up with find_link(). Any additional arguments specified are forwarded to this function.

问号(除其他外)是正则表达式 (regex) 元字符,因此如果您想将它们用于 follow_link/download_link,则需要对它们进行转义:

import re 
# ... 
browser.download_link(re.escape("https://www.ons.gov.uk/generator?format=xls&uri=/economy/grossdomesticproductgdp/timeseries/l2kq/qna")) 

但是,如果您访问的第一个页面不包含该直接链接,我不确定它是否有帮助。 (尽管先尝试。)

您可以使用浏览器的底层 requests session 来直接下载文件:

resp = browser.session.get("https://www.ons.gov.uk/generator?format=xls&uri=/economy/grossdomesticproductgdp/timeseries/l2kq/qna") 
resp.raise_for_status()  # raise an exception for 404, etc. 
with open('filename.xls', 'wb') as outf: 
  outf.write(resp.content)