Skip to main content
 首页 » 编程设计

python之Scrapy 或 BeautifulSoup 从各种网站上抓取链接和文本

2025年02月15日16thcjp

我试图从输入的 URL 中抓取链接,但它只适用于一个 url ( http://www.businessinsider.com )。它如何适应从输入的任何 url 中抓取?我正在使用 BeautifulSoup,但 Scrapy 更适合这个吗?

def WebScrape():   
    linktoenter = input('Where do you want to scrape from today?: ') 
    url = linktoenter 
    html = urllib.request.urlopen(url).read() 
    soup = BeautifulSoup(html, "lxml") 
 
    if linktoenter in url: 
        print('Retrieving your links...') 
        links = {} 
        n = 0 
        link_title=soup.findAll('a',{'class':'title'}) 
        n += 1 
        links[n] = link_title 
        for eachtitle in link_title: 
            print(eachtitle['href']+","+eachtitle.string) 
    else: 
        print('Please enter another Website...') 

请您参考如下方法:

您可以制作一个更通用的抓取工具,搜索所有标签和这些标签内的所有链接。获得所有链接的列表后,您可以使用正则表达式或类似表达式来查找与所需结构匹配的链接。

import requests 
from bs4 import BeautifulSoup 
import re 
 
response = requests.get('http://www.businessinsider.com') 
 
soup = BeautifulSoup(response.content) 
 
# find all tags 
tags = soup.find_all() 
 
links = [] 
 
# iterate over all tags and extract links 
for tag in tags: 
    # find all href links 
    tmp = tag.find_all(href=True) 
    # append masters links list with each link 
    map(lambda x: links.append(x['href']) if x['href'] else None, tmp) 
 
# example: filter only careerbuilder links 
filter(lambda x: re.search('[w]{3}\.careerbuilder\.com', x), links)