Skip to main content
 首页 » 编程设计

python:从任何网站提取文本

2024年10月25日9xing901022

到目前为止,我已经完成了我的工作,但它成功地从这两个网站获取了文本:

但我不知道我哪里做错了,它没有从其他网站获取文本,当我放置其他链接时它给我错误,例如:

错误:

Traceback (most recent call last):
File "C:\Users\DELL\Desktop\python\s\fyp\data extraction.py", line 20, in text = soup.select('.C_doc')[0].get_text() IndexError: list index out of range

我的代码:

import urllib 
from bs4 import BeautifulSoup 
url = "http://www.i-programmer.info/babbages-bag/477-trees.html" #unsuccessfull  
#url = "http://www.tutorialspoint.com/cplusplus/index.htm"   #doing successfully 
#url = "http://www.cplusplus.com/doc/tutorial/program_structure/" #doing successfully 
html = urllib.urlopen(url).read() 
soup = BeautifulSoup(html) 
 
# kill all script and style elements 
for script in soup(["script", "style","a","<div id=\"bottom\" >"]): 
    script.extract()    # rip it out 
 
# get text 
#text = soup.select('.C_doc')[0].get_text() 
#text = soup.select('.content')[0].get_text() 
 
if soup.select('.content'): 
    text = soup.select('.content')[0].get_text() 
else: 
    text = soup.select('.C_doc')[0].get_text() 
 
# break into lines and remove leading and trailing space on each 
lines = (line.strip() for line in text.splitlines()) 
# break multi-headlines into a line each 
chunks = (phrase.strip() for line in lines for phrase in line.split("  ")) 
# drop blank lines 
text = '\n'.join(chunk for chunk in chunks if chunk) 
 
print text 
fo = open('foo.txt', 'w') 
fo.seek(0, 2) 
line = fo.writelines( text ) 
fo.close() 
#writing done :) 

请您参考如下方法:

尝试使用

Text = soup.findAll(text=True) 

更新

这是一个基本的文本剥离器,您可以从中开始。

import urllib 
from bs4 import BeautifulSoup 
url = "http://www.i-programmer.info/babbages-bag/477-trees.html"  
html = urllib.urlopen(url).read() 
soup = BeautifulSoup(html) 
 
for script in soup(["script", "style","a","<div id=\"bottom\" >"]): 
    script.extract()     
 
text = soup.findAll(text=True) 
for p in text: 
    print p