Skip to main content
 首页 » 编程设计

python之使用 BeautifulSoup 解析 HTML 结构

2024年12月31日7落叶无声

这是我需要解析的 HTML 文档的结构(引用更新 3):

    <div id="txt_123_C01" style="position:absolute; left:5px; top:206px; width:532px; height:8912px;"> 
        <div class="Normal-P1"> 
        <span class="Normal-C2">Main title 1<br></span></div> 
        <div class="Normal-P1"> 
        <span class="Normal-C3">Optional Subtitle<br></span></div> 
        <div class="Normal-P1"> 
        <span class="Normal-C3">Second Optional Subtitle</span> 
        <span class="Normal-C4">Text blurb 1.<br></span></div> 
        <div class="Normal-P1"> 
        <span class="Normal-C4">Text blurb 2.<br></span></div> 
        <div class="Normal-P1"> 
        <span class="Normal-C4">Text blurb 4.<br></span></div> 
        <span class="Normal-C4"><br></span></div> 
        <div class="Normal-P1"> 
        <span class="Normal-C3"><br></span></div> 
 
    <div class="Normal-P1"> 
        <span class="Normal-C2">Main title 2<br></span></div> 
        <div class="Normal-P1"> 
        <span class="Normal-C3">Subtitle 1</span> 
        <span class="Normal-C4"> Other text blurb 1.<br></span></div> 
        <div class="Normal-P1"> 
        <span class="Normal-C4"> Other text blurb 2.<br></span></div> 

我想生成如下所示的 CSV:

    Main Title     Optional Subtitle 1     Optional Subtitle 2        Text Blurb 
    ----------     -------------------     -------------------       ------------------------      
    Main title 1   Optional Subtitle       Second Optional Subtitle   Text blurb1. Textblurb2. Text blurb 4. 
    Main Title 2     Subtitle 1                                         Other text blurb 2. 

我试过:

soup = BeautifulSoup(page,'xml') 
divText = soup.find_all('div', {'class':'Normal-P1'}) 
for item in divText: 
    spanTitle = soup.find_all('span',{'class':'Normal-C2'}) 
    spanOptopnal = soup.find_all('span',{'class':'Normal-C3'}) 

但是,这种方法不允许我分离出 Normal-P1我从 C2 开始的类(class)至 C4然后重新开始。 C3在 C4 之间和下一个C2并不总是存在。在这些情况下,C4是下一个 C2 之前的最后一个标签.

我考虑过把所有 div s 在列表中,然后根据 C2 将它们拆分为子列表处理它们。我想弄清楚是否有使用 bs4 的更优雅的解决方案。

更新 1

过一会儿再回到这个话题。我刚刚使用下面的答案检查了我的输出,发现了一个问题。

看着

   titles = soup.select(".Normal-P1 .Normal-C2") 
   for entry in titles: 
            print "entry:",entry 
            parent = entry.parent 
            print "parent: ",parent 
            subtitles = [ 
                subtitle.text for subtitle in 
                parent.select(' ~ .Normal-P1 .Normal-C3') 
            ] 
            print "subtitles:",subtitles 

我找到了 subtitles包含来自父级外部的结果(即所有 titles )。输出如下所示:

entry: <span class="Normal-C2">Main title 1<br/></span> 
parent:  <div class="Normal-P1"> 
<span class="Normal-C2">Main title 1<br/></span></div> 
subtitles: [Optional Subtitle,Second Optional Subtitle,Subtitle 1] 
 
 
entry: <span class="Normal-C2">Main title 2<br/></span> 
parent:  <div class="Normal-P1"> 
<span class="Normal-C2">Main title 2<br/></span></div> 
subtitles: [Subtitle 1] 

更新 2

parent.select(" ~ .Normal-P1 .Normal-C3")似乎是导致问题的原因。

问题出在解决方案提供的 HTML 中:<span class="Normal-C4"><br></span> </div> .不见了<div class="Normal-P1">和结束</div>在最后。在进行这些更改时,我在这个示例 HTML 中也看到了同样的问题(文档中的所有字幕都显示为一个条目)。

我仔细检查了缩进,我觉得没问题。我做错了什么?

更新 3

这是完整的HTML

<!DOCTYPE html> 
<html lang="en"> 
<head> 
  <meta charset="UTF-8"> 
  <title>Hello World</title> 
</head> 
<body> 
  <div id="txt_123_C01" style="position:absolute; left:5px; top:206px; width:532px; height:8912px;"> 
    <div class="Normal-P1"> 
      <span class="Normal-C2">Main title 1<br></span> 
    </div> 
    <div class="Normal-P1"> 
      <span class="Normal-C3">Optional Subtitle<br></span> 
    </div> 
    <div class="Normal-P1"> 
      <span class="Normal-C3">Second Optional Subtitle</span> 
      <span class="Normal-C4">Text blurb 1.<br></span> 
    </div> 
    <div class="Normal-P1"> 
      <span class="Normal-C4">Text blurb 2.<br></span> 
    </div> 
    <div class="Normal-P1"> 
      <span class="Normal-C4">Text blurb 4.<br></span> 
    </div> 
    <div class="Normal-P1"> 
    <span class="Normal-C4"><br></span> 
  </div> 
  <div class="Normal-P1"> 
    <span class="Normal-C3"><br></span> 
  </div> 
 
  <div class="Normal-P1"> 
    <span class="Normal-C2">Main title 2<br></span> 
  </div> 
  <div class="Normal-P1"> 
    <span class="Normal-C3">New Subtitle 1</span> 
    <span class="Normal-C4"> Other text blurb 1.<br></span> 
  </div> 
  <div class="Normal-P1"> 
    <span class="Normal-C4"> Other text blurb 2.<br></span> 
  </div> 
</div> 
</body> 
</html> 

这是我看到的输出

    entry: <span class="Normal-C2">Main title 1<br/></span> 
    parent:  <div class="Normal-P1"> 
 
<span class="Normal-C2">Main title 1<br/></span> 
 
    </div> 
    subtitle: <span class="Normal-C3">Optional Subtitle<br/></span> 
    subtitle: <span class="Normal-C3">Second Optional Subtitle</span> 
    subtitle: <span class="Normal-C3"><br/></span> 
    subtitle: <span class="Normal-C3">New Subtitle 1</span> 
    entry: <span class="Normal-C2">Main title 2<br/></span> 
    parent:  <div class="Normal-P1"> 
    <span class="Normal-C2">Main title 2<br/></span> 
    </div> 
    subtitle: <span class="Normal-C3">New Subtitle 1</span> 

这是我目前的代码:

file = filepath + "test-page.html" 
parser = HTMLParser.HTMLParser() 
pageFile = codecs.open(file, 'r', encoding='utf-8') 
pageRaw = pageFile.read() 
page = parser.unescape(pageRaw) 
 
soup = bs4.BeautifulSoup(page,'lxml') 
titles = soup.select(".Normal-P1 .Normal-C2") 
 
for entry in titles: 
    print "entry:",entry 
    parent = entry.parent 
    print "parent: ",parent 
 
    for subtitle in parent.select(" ~ .Normal-P1 .Normal-C3"): 
        print "subtitle:", subtitle 

请您参考如下方法:

使用 CSS 选择器,您将要定位类名,您通过 .Class-Name 和 sibling ,这是通过 ParentTag ~ .Child-Class

Mozillas MDN Web Docs对它们有一个很好的入门。

python 文件:

import bs4 
import csv 
 
entries = [] 
 
with open("example.html", "r") as page: 
    soup = bs4.BeautifulSoup(page, 'lxml') 
 
    # CSS Selectors for items with class Normal-P1 followed by 
    # Normal-C2 
    titles = soup.select(".Normal-P1 .Normal-C2") 
 
    for entry in titles: 
        entry_dict = { 
            'Main Title': '', 
            'Optional Subtitle 1': '', 
            'Optional Subtitle 2': '', 
            'Text Blurb': '' 
        } 
        parent = entry.parent 
 
        entry_dict['Main Title'] = entry.text 
 
        subtitles = [ 
            subtitle.text for subtitle in 
            parent.select(' ~ .Normal-P1 .Normal-C3') 
            # CSS Selector for siblings of the same parent element that have 
            # classes Normal-P1 followed by Normal-C3 
        ] 
        try: 
            entry_dict['Optional Subtitle 1'] = subtitles[0] 
            entry_dict['Optional Subtitle 2'] = subtitles[1] 
        except IndexError: 
            pass 
 
        entry_dict['Text Blurb'] = ' '.join( 
            blurb.text for blurb in 
            parent.select(' ~ .Normal-P1 .Normal-C4') 
            # CSS Selector for siblings of the same parent element that have 
            # classes Normal-P1 followed by Normal-C4 
        ) 
 
        entries.append(entry_dict) 
 
    with open('out.csv', 'w') as csv_file: 
        fieldnames = [ 
            'Main Title', 
            'Optional Subtitle 1', 
            'Optional Subtitle 2', 
            'Text Blurb' 
        ] 
        writer = csv.DictWriter( 
            csv_file, 
            fieldnames=fieldnames, 
            quoting=csv.QUOTE_ALL, 
        ) 
        writer.writeheader() 
        for entry in entries: 
            writer.writerow(entry) 

使用的 html 文件:

<!DOCTYPE html> 
<html lang="en"> 
<head> 
  <meta charset="UTF-8"> 
  <title>Hello World</title> 
</head> 
<body> 
  <div id="txt_123_C01" style="position:absolute; left:5px; top:206px; width:532px; height:8912px;"> 
    <div class="Normal-P1"> 
      <span class="Normal-C2">Main title 1<br></span> 
    </div> 
    <div class="Normal-P1"> 
      <span class="Normal-C3">Optional Subtitle<br></span> 
    </div> 
    <div class="Normal-P1"> 
      <span class="Normal-C3">Second Optional Subtitle</span> 
      <span class="Normal-C4">Text blurb 1.<br></span> 
    </div> 
    <div class="Normal-P1"> 
      <span class="Normal-C4">Text blurb 2.<br></span> 
    </div> 
    <div class="Normal-P1"> 
      <span class="Normal-C4">Text blurb 4.<br></span> 
    </div> 
    <span class="Normal-C4"><br></span> 
  </div> 
  <div class="Normal-P1"> 
    <span class="Normal-C3"><br></span> 
  </div> 
 
  <div class="Normal-P1"> 
    <span class="Normal-C2">Main title 2<br></span> 
  </div> 
  <div class="Normal-P1"> 
    <span class="Normal-C3">Subtitle 1</span> 
    <span class="Normal-C4"> Other text blurb 1.<br></span> 
  </div> 
  <div class="Normal-P1"> 
    <span class="Normal-C4"> Other text blurb 2.<br></span> 
  </div> 
</body> 
</html>

csv 输出:

"Main Title","Optional Subtitle 1","Optional Subtitle 2","Text Blurb" 
 
"Main title 1","Optional Subtitle","Second Optional Subtitle","Text blurb 1. Text blurb 2. Text blurb 4." 
 
"Main title 2","Subtitle 1",""," Other text blurb 1.  Other text blurb 2." 

编辑: 我不确定你在用 HTMLParser 做什么,但没有必要这样做。 BeautifulSoup 可以很好地处理读取文件。

import bs4 
import codecs 
 
 
with codecs.open("example.html", "r", encoding='utf-8') as page: 
    soup = bs4.BeautifulSoup(page, 'lxml') 
 
    # CSS Selectors for items with class Normal-P1 followed by 
    # Normal-C2 
    titles = soup.select(".Normal-P1 .Normal-C2") 
 
    for entry in titles: 
        print("entry: ", entry.text) 
        parent = entry.parent 
        print("parent:", parent) 
 
        subtitles = [ 
            subtitle.text for subtitle in 
            parent.select(' ~ .Normal-P1 .Normal-C3') 
            # CSS Selector for siblings of the same parent element that have 
            # classes Normal-P1 followed by Normal-C3 
        ] 
        try: 
            print("subtitle: ", subtitles[0]) 
            print('subtitle: ', subtitles[1]) 
        except IndexError: 
            pass 

输出

entry:  Main title 1 
parent: <div class="Normal-P1"> 
<span class="Normal-C2">Main title 1<br/></span> 
</div> 
subtitle:  Optional Subtitle 
subtitle:  Second Optional Subtitle 
entry:  Main title 2 
parent: <div class="Normal-P1"> 
<span class="Normal-C2">Main title 2<br/></span> 
</div> 
subtitle:  Subtitle 1