这是我需要解析的 HTML 文档的结构(引用更新 3):
<div id="txt_123_C01" style="position:absolute; left:5px; top:206px; width:532px; height:8912px;">
<div class="Normal-P1">
<span class="Normal-C2">Main title 1<br></span></div>
<div class="Normal-P1">
<span class="Normal-C3">Optional Subtitle<br></span></div>
<div class="Normal-P1">
<span class="Normal-C3">Second Optional Subtitle</span>
<span class="Normal-C4">Text blurb 1.<br></span></div>
<div class="Normal-P1">
<span class="Normal-C4">Text blurb 2.<br></span></div>
<div class="Normal-P1">
<span class="Normal-C4">Text blurb 4.<br></span></div>
<span class="Normal-C4"><br></span></div>
<div class="Normal-P1">
<span class="Normal-C3"><br></span></div>
<div class="Normal-P1">
<span class="Normal-C2">Main title 2<br></span></div>
<div class="Normal-P1">
<span class="Normal-C3">Subtitle 1</span>
<span class="Normal-C4"> Other text blurb 1.<br></span></div>
<div class="Normal-P1">
<span class="Normal-C4"> Other text blurb 2.<br></span></div>
我想生成如下所示的 CSV:
Main Title Optional Subtitle 1 Optional Subtitle 2 Text Blurb
---------- ------------------- ------------------- ------------------------
Main title 1 Optional Subtitle Second Optional Subtitle Text blurb1. Textblurb2. Text blurb 4.
Main Title 2 Subtitle 1 Other text blurb 2.
我试过:
soup = BeautifulSoup(page,'xml')
divText = soup.find_all('div', {'class':'Normal-P1'})
for item in divText:
spanTitle = soup.find_all('span',{'class':'Normal-C2'})
spanOptopnal = soup.find_all('span',{'class':'Normal-C3'})
但是,这种方法不允许我分离出 Normal-P1我从 C2 开始的类(class)至 C4然后重新开始。 C3在 C4 之间和下一个C2并不总是存在。在这些情况下,C4是下一个 C2 之前的最后一个标签.
我考虑过把所有 div s 在列表中,然后根据 C2 将它们拆分为子列表处理它们。我想弄清楚是否有使用 bs4 的更优雅的解决方案。
更新 1
过一会儿再回到这个话题。我刚刚使用下面的答案检查了我的输出,发现了一个问题。
看着
titles = soup.select(".Normal-P1 .Normal-C2")
for entry in titles:
print "entry:",entry
parent = entry.parent
print "parent: ",parent
subtitles = [
subtitle.text for subtitle in
parent.select(' ~ .Normal-P1 .Normal-C3')
]
print "subtitles:",subtitles
我找到了 subtitles包含来自父级外部的结果(即所有 titles )。输出如下所示:
entry: <span class="Normal-C2">Main title 1<br/></span>
parent: <div class="Normal-P1">
<span class="Normal-C2">Main title 1<br/></span></div>
subtitles: [Optional Subtitle,Second Optional Subtitle,Subtitle 1]
entry: <span class="Normal-C2">Main title 2<br/></span>
parent: <div class="Normal-P1">
<span class="Normal-C2">Main title 2<br/></span></div>
subtitles: [Subtitle 1]
更新 2
parent.select(" ~ .Normal-P1 .Normal-C3")似乎是导致问题的原因。
问题出在解决方案提供的 HTML 中:<span class="Normal-C4"><br></span> </div> .不见了<div class="Normal-P1">和结束</div>在最后。在进行这些更改时,我在这个示例 HTML 中也看到了同样的问题(文档中的所有字幕都显示为一个条目)。
我仔细检查了缩进,我觉得没问题。我做错了什么?
更新 3
这是完整的HTML
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Hello World</title>
</head>
<body>
<div id="txt_123_C01" style="position:absolute; left:5px; top:206px; width:532px; height:8912px;">
<div class="Normal-P1">
<span class="Normal-C2">Main title 1<br></span>
</div>
<div class="Normal-P1">
<span class="Normal-C3">Optional Subtitle<br></span>
</div>
<div class="Normal-P1">
<span class="Normal-C3">Second Optional Subtitle</span>
<span class="Normal-C4">Text blurb 1.<br></span>
</div>
<div class="Normal-P1">
<span class="Normal-C4">Text blurb 2.<br></span>
</div>
<div class="Normal-P1">
<span class="Normal-C4">Text blurb 4.<br></span>
</div>
<div class="Normal-P1">
<span class="Normal-C4"><br></span>
</div>
<div class="Normal-P1">
<span class="Normal-C3"><br></span>
</div>
<div class="Normal-P1">
<span class="Normal-C2">Main title 2<br></span>
</div>
<div class="Normal-P1">
<span class="Normal-C3">New Subtitle 1</span>
<span class="Normal-C4"> Other text blurb 1.<br></span>
</div>
<div class="Normal-P1">
<span class="Normal-C4"> Other text blurb 2.<br></span>
</div>
</div>
</body>
</html>
这是我看到的输出
entry: <span class="Normal-C2">Main title 1<br/></span>
parent: <div class="Normal-P1">
<span class="Normal-C2">Main title 1<br/></span>
</div>
subtitle: <span class="Normal-C3">Optional Subtitle<br/></span>
subtitle: <span class="Normal-C3">Second Optional Subtitle</span>
subtitle: <span class="Normal-C3"><br/></span>
subtitle: <span class="Normal-C3">New Subtitle 1</span>
entry: <span class="Normal-C2">Main title 2<br/></span>
parent: <div class="Normal-P1">
<span class="Normal-C2">Main title 2<br/></span>
</div>
subtitle: <span class="Normal-C3">New Subtitle 1</span>
这是我目前的代码:
file = filepath + "test-page.html"
parser = HTMLParser.HTMLParser()
pageFile = codecs.open(file, 'r', encoding='utf-8')
pageRaw = pageFile.read()
page = parser.unescape(pageRaw)
soup = bs4.BeautifulSoup(page,'lxml')
titles = soup.select(".Normal-P1 .Normal-C2")
for entry in titles:
print "entry:",entry
parent = entry.parent
print "parent: ",parent
for subtitle in parent.select(" ~ .Normal-P1 .Normal-C3"):
print "subtitle:", subtitle
请您参考如下方法:
使用 CSS 选择器,您将要定位类名,您通过 .Class-Name 和 sibling ,这是通过 ParentTag ~ .Child-Class
Mozillas MDN Web Docs对它们有一个很好的入门。
python 文件:
import bs4
import csv
entries = []
with open("example.html", "r") as page:
soup = bs4.BeautifulSoup(page, 'lxml')
# CSS Selectors for items with class Normal-P1 followed by
# Normal-C2
titles = soup.select(".Normal-P1 .Normal-C2")
for entry in titles:
entry_dict = {
'Main Title': '',
'Optional Subtitle 1': '',
'Optional Subtitle 2': '',
'Text Blurb': ''
}
parent = entry.parent
entry_dict['Main Title'] = entry.text
subtitles = [
subtitle.text for subtitle in
parent.select(' ~ .Normal-P1 .Normal-C3')
# CSS Selector for siblings of the same parent element that have
# classes Normal-P1 followed by Normal-C3
]
try:
entry_dict['Optional Subtitle 1'] = subtitles[0]
entry_dict['Optional Subtitle 2'] = subtitles[1]
except IndexError:
pass
entry_dict['Text Blurb'] = ' '.join(
blurb.text for blurb in
parent.select(' ~ .Normal-P1 .Normal-C4')
# CSS Selector for siblings of the same parent element that have
# classes Normal-P1 followed by Normal-C4
)
entries.append(entry_dict)
with open('out.csv', 'w') as csv_file:
fieldnames = [
'Main Title',
'Optional Subtitle 1',
'Optional Subtitle 2',
'Text Blurb'
]
writer = csv.DictWriter(
csv_file,
fieldnames=fieldnames,
quoting=csv.QUOTE_ALL,
)
writer.writeheader()
for entry in entries:
writer.writerow(entry)
使用的 html 文件:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Hello World</title>
</head>
<body>
<div id="txt_123_C01" style="position:absolute; left:5px; top:206px; width:532px; height:8912px;">
<div class="Normal-P1">
<span class="Normal-C2">Main title 1<br></span>
</div>
<div class="Normal-P1">
<span class="Normal-C3">Optional Subtitle<br></span>
</div>
<div class="Normal-P1">
<span class="Normal-C3">Second Optional Subtitle</span>
<span class="Normal-C4">Text blurb 1.<br></span>
</div>
<div class="Normal-P1">
<span class="Normal-C4">Text blurb 2.<br></span>
</div>
<div class="Normal-P1">
<span class="Normal-C4">Text blurb 4.<br></span>
</div>
<span class="Normal-C4"><br></span>
</div>
<div class="Normal-P1">
<span class="Normal-C3"><br></span>
</div>
<div class="Normal-P1">
<span class="Normal-C2">Main title 2<br></span>
</div>
<div class="Normal-P1">
<span class="Normal-C3">Subtitle 1</span>
<span class="Normal-C4"> Other text blurb 1.<br></span>
</div>
<div class="Normal-P1">
<span class="Normal-C4"> Other text blurb 2.<br></span>
</div>
</body>
</html>
csv 输出:
"Main Title","Optional Subtitle 1","Optional Subtitle 2","Text Blurb"
"Main title 1","Optional Subtitle","Second Optional Subtitle","Text blurb 1. Text blurb 2. Text blurb 4."
"Main title 2","Subtitle 1",""," Other text blurb 1. Other text blurb 2."
编辑: 我不确定你在用 HTMLParser 做什么,但没有必要这样做。 BeautifulSoup 可以很好地处理读取文件。
import bs4
import codecs
with codecs.open("example.html", "r", encoding='utf-8') as page:
soup = bs4.BeautifulSoup(page, 'lxml')
# CSS Selectors for items with class Normal-P1 followed by
# Normal-C2
titles = soup.select(".Normal-P1 .Normal-C2")
for entry in titles:
print("entry: ", entry.text)
parent = entry.parent
print("parent:", parent)
subtitles = [
subtitle.text for subtitle in
parent.select(' ~ .Normal-P1 .Normal-C3')
# CSS Selector for siblings of the same parent element that have
# classes Normal-P1 followed by Normal-C3
]
try:
print("subtitle: ", subtitles[0])
print('subtitle: ', subtitles[1])
except IndexError:
pass
输出
entry: Main title 1
parent: <div class="Normal-P1">
<span class="Normal-C2">Main title 1<br/></span>
</div>
subtitle: Optional Subtitle
subtitle: Second Optional Subtitle
entry: Main title 2
parent: <div class="Normal-P1">
<span class="Normal-C2">Main title 2<br/></span>
</div>
subtitle: Subtitle 1
