Skip to main content
 首页 » 编程设计

python之如何使用 Beautifulsoup 访问前五个谷歌结果链接

2025年02月15日9shangdawei

我想访问来自 Google 的前五个(或任何指定数量的)结果链接。通过研究,我发现并修改了如下代码。

import requests 
from bs4 import BeautifulSoup 
import re     
search = raw_input("Search:") 
page = requests.get("https://www.google.com/search?q=" + search) 
soup = BeautifulSoup(page.content, "lxml") 
links = soup.find("a") 
print links.get('href') 

这将返回页面上的第一个链接,似乎每次都是 Google 图片选项卡。

这不完全是我想要的。对于初学者,我不想要任何谷歌网站的链接,只想要结果。另外,我想要前三个或五个或任何指定数量的结果。

我如何使用 python 来执行此操作?

提前致谢!

请您参考如下方法:

您可以使用:

import requests 
from bs4 import BeautifulSoup 
import re 
search = input("Search:") 
results = 100 # valid options 10, 20, 30, 40, 50, and 100 
page = requests.get(f"https://www.google.com/search?q={search}&num={results}") 
soup = BeautifulSoup(page.content, "html5lib") 
links = soup.findAll("a") 
for link in links : 
    link_href = link.get('href') 
    if "url?q=" in link_href and not "webcache" in link_href: 
        print (link.get('href').split("?q=")[1].split("&sa=U")[0]) 

Google Search Demo

对于 duckduckgo.com 使用:

import requests 
from bs4 import BeautifulSoup 
import re 
search = input("Search:") 
h = {"Host":"duckduckgo.com", "Origin": "https://duckduckgo.com", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0"} 
d = {"q":search} 
page = requests.post(f"https://duckduckgo.com/html/", data=d, headers=h) 
soup = BeautifulSoup(page.content, "html5lib") 
links = soup.findAll("a", {"class": "result__a"}) 
for link in links : 
    link_href = link.get('href') 
    if not "https://duckduckgo.com" in link_href: 
        print(link_href)