Skip to main content
 首页 » 编程设计

python之如何用漂亮的汤按标题提取网址

2025年02月15日8softidea

我有一个我感兴趣的链接列表:

lis = ['https://example1.com', 'https://example2.com', ..., 'https://exampleN.com'] 

在那些链接中有几个 url,我想提取一些特定的内部 url。此类网址具有以下形式:

<a href="https://interesting-linkN.com" target="_blank" title="Url to news"> News JPG </a>

如何检查 lis 的所有元素?并返回 lis的访问过的链接,只有标题为 Url to news 的网址在 Pandas 数据框中?,像这样 (**):

visited_link, extracted_link 
https://www.example1.com, NaN 
https://www.example2.com, NaN 
https://www.example3.com, https://interesting-linkN.com 

请注意,对于 lis 的元素谁没有任何<a href="https://interesting-linkN.com" target="_blank" title="Url to news"> News JPG </a>我想返回 NaN .

我试过了 this和:

def extract_jpg_url(a_link): 
    page = requests.get(a_link) 
    tree = html.fromstring(page.content) 
    # here is the problem... not all interesting links have this xpath, how can I select by title? 
    #(apparently all the jpg urls have this form: title="Url to news") 
    interesting_link = tree.xpath(".//*[@id='object']//tbody//tr//td//span//a/@href") 
    if len(interesting_link) == 0: 
        return'NaN' 
    else: 
        return 'image link ', interesting_link 
then: 
 
    df['news_link'] = df['urls_from_lis'].apply(extract_jpg_url) 

但是,后一种方法耗时太长,而且不是lis的所有元素。匹配给定的 xpath(检查评论)我能得到什么的任何想法(**)?

请您参考如下方法:

这不会完全返回您想要的结果 (NaN),但它会让您大致了解如何简单高效地完成这项工作。

from bs4 import BeautifulSoup 
from multiprocessing.pool import ThreadPool 
import requests 
 
def extract_urls(link): 
    r = requests.get(link) 
    html = r.text 
    soup = BeautifulSoup(html, "html.parser") 
    results = soup.findAll('a', {'title': 'Url to news'}) 
    results = [x['href'] for x in results] 
    return (link, results) 
 
links = [ 
    "https://example1.com", 
    "https://example2.com", 
    "https://exampleN.com", ] 
 
p = ThreadPool(10) 
r = p.map(extract_urls, links) 
 
for url, results in r: 
    print(url, results)