Skip to main content
 首页 » 编程设计

Python BeautifulSoup 获取图像、文本和 URL

2024年11月24日7落叶无声

我正在尝试从以下网址(http://www.ancient-hebrew.org/m/dictionary/1000.html)抓取数据。所以每个希伯来语单词部分都以 img urls 开头,后跟 2 个文本,即实际的希伯来语单词及其发音。例如,url 中的第一个条目是以下“img1 img2 img3 אֶלֶף e-leph”,使用 wget 下载 html 后,希伯来语单词是一个 unicode

例如,我提供的以下代码 <img src="../../files/heb-anc-sm-pey.jpg"/>和 <font face="arial" size="+1"> unicode_hebrew_text </font>和 <a href="audio/ 505 .mp3"><img border="0" height="25" src="../../files/icon_audio.gif" width="25"/></a>

相反,我只想要 ../../files/heb-anc-sm-pey.jpg 和 unicode_hebrew_text和 audio/505.mp3 (without any spaces in between)

from bs4 import BeautifulSoup 
 
raw_html = open('/Users/gansaikhanshur/TESTING/webScraping/1000.html').read() 
html = BeautifulSoup(raw_html, 'html.parser') 
 
# output: <img src="../../files/heb-anc-sm-pey.jpg"/> 
imgs = html.findAll("img") 
for image in imgs: 
    # print image source 
    if "jpg" in str(image): 
        print(image) 
 
# output: <font face="arial" size="+1"> unicode_hebrew_text </font> 
font = html('font', face="arial", size="+1") 
for f in font: 
    continue 
 
# output: <a href="audio/   505 .mp3"><img border="0" height="25" src="../../files/icon_audio.gif" width="25"/></a> 
mp3file = html.findAll(href=True) 
for mp3 in mp3file: 
    if "mp3" in str(mp3): 
        continue 

如您所见,我的代码并没有真正完成这项工作。最后,我想获取 URL 中每个单词的信息,并将其保存为文本文件或 json 文件,以较容易的为准。

例如,图片:URLsOfImages,希伯来语单词:txt,发音:txt,URLtoAudio:txt

以及下一个词等等。

请您参考如下方法:

您要抓取的网页有点棘手。

现在,我为什么这么说?问题是,网页并没有大量使用 div 标签或类似的东西来将单个希伯来语单词及其相关数据放在一个 block 中(让我称希伯来语单词及其相关数据为希伯来语 block )。但是,如果您关注页面源代码,那么您会注意到每个希伯来语 block 都从一个 anchor 标记开始,看起来像这样:<a name="505"></a> .值“505”将根据每个希伯来语 block 的强数而变化。所以,我试着把它作为每个希伯来语 block 的起点。现在在希伯来语 block 中,您可以使用扬声器图标 <a href="audio/ 505 .mp3"><img src="../../files/icon_audio.gif" width="25" height="25" border="0"></a>来表示希伯来语 block 的终点。

考虑到上述想法,我想到了这个:

from bs4 import BeautifulSoup 
from bs4 import NavigableString, Tag  
import requests 
import re 
 
def make_soup(url): 
    html = requests.get(url, verify = False) 
    soup = BeautifulSoup(html.text, 'html.parser') 
    return soup 
 
 
url = "http://www.ancient-hebrew.org/m/dictionary/1000.html" 
soup = make_soup(url) 
 
points = soup.find_all("a", attrs = {"name" : re.compile("[0-9]+")})  
#This will help you locate the starting points of the Hebrew block 
 
for point in points: 
    point_name = point['name']  
    images = [] #Will hold the links for the Old Hebrew images 
    text = "" #Will hold the immediate next Unicode Hebrew text 
    usage = "" #Will hold the usage/meaning data 
    mp3 = "" #Will hold the link to audio file 
 
    temp = point.next_sibling  
    #'temp' variable will be used as a navigator to extract the data within the Hebrew block  
 
    while(True): 
        #If 'temp' is a HTML 'Tag' by nature, then it'll contain the Unicode Hebrew text and links for images& audio 
        if(type(temp) is Tag):  
            if(temp.name == 'img'):         
                #If it's an 'img' tag, then put it to 'images' list  
                images.append(temp['src']) 
            elif(temp.name == 'font'): 
                #If it's a 'font' tag, then put it in 'text' 
                text = temp.string 
            elif(temp.name == 'a'): 
                #If it's 'a' tag, then it contains the link to audio file 
                mp3 = temp['href']   
                #Link to audio file will mark the end of the Hebrew block. So, 'break' out 
                break 
        elif(type(temp) is NavigableString): 
            #If 'temp' acquires a 'NavigableString' instead of a 'Tag' then it's usage/meaning 
            usage += temp 
        #Move to the next item within the Hebrew block 
        temp = temp.next_sibling 
 
    print([point_name, images, mp3, text, usage])