我有一个我感兴趣的链接列表:
lis = ['https://example1.com', 'https://example2.com', ..., 'https://exampleN.com']
在那些链接中有几个 url,我想提取一些特定的内部 url。此类网址具有以下形式:
<a href="https://interesting-linkN.com" target="_blank" title="Url to news"> News JPG </a>
如何检查 lis 的所有元素?并返回 lis的访问过的链接,只有标题为 Url to news 的网址在 Pandas 数据框中?,像这样 (**):
visited_link, extracted_link
https://www.example1.com, NaN
https://www.example2.com, NaN
https://www.example3.com, https://interesting-linkN.com
请注意,对于 lis 的元素谁没有任何<a href="https://interesting-linkN.com" target="_blank" title="Url to news"> News JPG </a>我想返回 NaN .
我试过了 this和:
def extract_jpg_url(a_link):
page = requests.get(a_link)
tree = html.fromstring(page.content)
# here is the problem... not all interesting links have this xpath, how can I select by title?
#(apparently all the jpg urls have this form: title="Url to news")
interesting_link = tree.xpath(".//*[@id='object']//tbody//tr//td//span//a/@href")
if len(interesting_link) == 0:
return'NaN'
else:
return 'image link ', interesting_link
then:
df['news_link'] = df['urls_from_lis'].apply(extract_jpg_url)
但是,后一种方法耗时太长,而且不是lis的所有元素。匹配给定的 xpath(检查评论)我能得到什么的任何想法(**)?
请您参考如下方法:
这不会完全返回您想要的结果 (NaN),但它会让您大致了解如何简单高效地完成这项工作。
from bs4 import BeautifulSoup
from multiprocessing.pool import ThreadPool
import requests
def extract_urls(link):
r = requests.get(link)
html = r.text
soup = BeautifulSoup(html, "html.parser")
results = soup.findAll('a', {'title': 'Url to news'})
results = [x['href'] for x in results]
return (link, results)
links = [
"https://example1.com",
"https://example2.com",
"https://exampleN.com", ]
p = ThreadPool(10)
r = p.map(extract_urls, links)
for url, results in r:
print(url, results)
