Skip to main content
 首页 » 编程设计

Python BeautifulSoup 挑选网页,相同的代码工作和关闭

2024年11月01日9over140

我正在使用相同的代码来获取网络文本,但大多数时候它显示“警告:根:某些字符无法解码,并被替换为替换字符。”,令人惊讶的是有时它有效,例如我运行代码12次,1次成功。

相同的代码,相同的网址。为什么会这样?

from bs4 import BeautifulSoup 
import re 
import urllib2 
 
 
url = "http://nz.sports.search.yahoo.com/search?p=basketball&fr=sports-nz-ss&age=1w&focuslim=age" 
page = urllib2.urlopen(url) 
soup = BeautifulSoup(page.read()) 
 
web_p = soup.find_all('span',class_='url') 
 
for web in web_p: 
    print web  

引用详情如下:

Traceback (most recent call last): 
  File "C:\Python27\lib\idlelib\run.py", line 112, in main 
seq, request = rpc.request_queue.get(block=True, timeout=0.05) 
  File "C:\Python27\lib\Queue.py", line 176, in get 
    raise Empty 
Empty 

请您参考如下方法:

感谢 isedev 的指导和 Does python urllib2 automatically uncompress gzip data fetched from webpage? 中的答案,代码更改为以下工作:

from StringIO import StringIO 
import gzip 
from bs4 import BeautifulSoup 
import re 
import urllib2 
 
 
request = urllib2.Request('http://nz.sports.search.yahoo.com/search?p=basketball&fr=sports-nz-ss&age=1w&focuslim=age') 
request.add_header('Accept-encoding', 'gzip') 
response = urllib2.urlopen(request) 
if response.info().get('Content-Encoding') == 'gzip': 
    buf = StringIO( response.read()) 
    f = gzip.GzipFile(fileobj=buf) 
    data = f.read() 
 
soup = BeautifulSoup(data) 
 
web_p = soup.find_all('span',class_='url') 
 
for web in web_p: 
    print web 


感谢 Blender 的指导,代码可以简化:

from bs4 import BeautifulSoup 
import requests 
 
html = requests.get('http://nz.sports.search.yahoo.com/search?p=basketball&fr=sports-nz-ss&age=1w&focuslim=age').text 
soup = BeautifulSoup(html) 
web_p = soup.find_all('span',class_='url') 
for web in web_p: 
    print web