Skip to main content
 首页 » 编程设计

python之网络爬虫递归 BeautifulSoup

2025年01月19日9zengkefu

我正在尝试以递归方式抓取所有英文文章链接的维基百科网址。我想对 n 执行深度优先遍历,但由于某种原因,我的代码不会在每次遍历时重复出现。知道为什么吗?

def crawler(url, depth): 
    if depth == 0: 
        return None 
    links = bs.find("div",{"id" : "bodyContent"}).findAll("a" , href=re.compile("(/wiki/)+([A-Za-z0-9_:()])+")) 
 
    print ("Level ",depth," ",url) 
    for link in links: 
        if ':' not in link['href']: 
            crawler("https://en.wikipedia.org"+link['href'], depth - 1) 

这是对爬虫的调用

url = "https://en.wikipedia.org/wiki/Harry_Potter" 
html = urlopen(url) 
bs = BeautifulSoup(html, "html.parser") 
crawler(url,3) 

请您参考如下方法:

您需要获取每个 不同网址的页面源(向页面发送请求)。您在 crawler() 函数中缺少该部分。在函数外添加这些行,不会递归调用它们。

def crawler(url, depth): 
    if depth == 0: 
        return None 
 
    html = urlopen(url)                        # You were missing  
    soup = BeautifulSoup(html, 'html.parser')  # these lines. 
 
    links = soup.find("div",{"id" : "bodyContent"}).findAll("a", href=re.compile("(/wiki/)+([A-Za-z0-9_:()])+")) 
 
    print("Level ", depth, url) 
    for link in links: 
        if ':' not in link['href']: 
            crawler("https://en.wikipedia.org"+link['href'], depth - 1) 
 
url = "https://en.wikipedia.org/wiki/Big_data" 
crawler(url, 3) 

部分输出:

Level  3 https://en.wikipedia.org/wiki/Big_data 
Level  2 https://en.wikipedia.org/wiki/Big_Data_(band) 
Level  1 https://en.wikipedia.org/wiki/Brooklyn 
Level  1 https://en.wikipedia.org/wiki/Electropop 
Level  1 https://en.wikipedia.org/wiki/Alternative_dance 
Level  1 https://en.wikipedia.org/wiki/Indietronica 
Level  1 https://en.wikipedia.org/wiki/Indie_rock 
Level  1 https://en.wikipedia.org/wiki/Warner_Bros._Records 
Level  1 https://en.wikipedia.org/wiki/Joywave 
Level  1 https://en.wikipedia.org/wiki/Electronic_music 
Level  1 https://en.wikipedia.org/wiki/Dangerous_(Big_Data_song) 
Level  1 https://en.wikipedia.org/wiki/Joywave 
Level  1 https://en.wikipedia.org/wiki/Billboard_(magazine) 
Level  1 https://en.wikipedia.org/wiki/Alternative_Songs 
Level  1 https://en.wikipedia.org/wiki/2.0_(Big_Data_album)