我正在使用相同的代码来获取网络文本,但大多数时候它显示“警告:根:某些字符无法解码,并被替换为替换字符。”,令人惊讶的是有时它有效,例如我运行代码12次,1次成功。
相同的代码,相同的网址。为什么会这样?
from bs4 import BeautifulSoup
import re
import urllib2
url = "http://nz.sports.search.yahoo.com/search?p=basketball&fr=sports-nz-ss&age=1w&focuslim=age"
page = urllib2.urlopen(url)
soup = BeautifulSoup(page.read())
web_p = soup.find_all('span',class_='url')
for web in web_p:
print web
引用详情如下:
Traceback (most recent call last):
File "C:\Python27\lib\idlelib\run.py", line 112, in main
seq, request = rpc.request_queue.get(block=True, timeout=0.05)
File "C:\Python27\lib\Queue.py", line 176, in get
raise Empty
Empty
请您参考如下方法:
感谢 isedev 的指导和 Does python urllib2 automatically uncompress gzip data fetched from webpage? 中的答案,代码更改为以下工作:
from StringIO import StringIO
import gzip
from bs4 import BeautifulSoup
import re
import urllib2
request = urllib2.Request('http://nz.sports.search.yahoo.com/search?p=basketball&fr=sports-nz-ss&age=1w&focuslim=age')
request.add_header('Accept-encoding', 'gzip')
response = urllib2.urlopen(request)
if response.info().get('Content-Encoding') == 'gzip':
buf = StringIO( response.read())
f = gzip.GzipFile(fileobj=buf)
data = f.read()
soup = BeautifulSoup(data)
web_p = soup.find_all('span',class_='url')
for web in web_p:
print web
感谢 Blender 的指导,代码可以简化:
from bs4 import BeautifulSoup
import requests
html = requests.get('http://nz.sports.search.yahoo.com/search?p=basketball&fr=sports-nz-ss&age=1w&focuslim=age').text
soup = BeautifulSoup(html)
web_p = soup.find_all('span',class_='url')
for web in web_p:
print web
