Skip to main content
 首页 » 编程设计

python之当我使用 selenium 抓取网站时得到 python UnicodeEncodeError

2024年10月25日11qq78292959

我正在尝试使用 selenium 来抓取该网站上的论文标题:http://www.ncbi.nlm.nih.gov/pubmed?term=(%222013%22%5BDate%20-%20Publication%5D%20%3A%20%222013%22%5BDate%20-%20Publication%5D)

#coding="utf-8" 
 
from selenium import webdriver 
from selenium.webdriver.common.keys import Keys 
import time 
 
domain = "http://www.ncbi.nlm.nih.gov/" 
url_tail = "pubmed?term=(%222013%22%5BDate%20-%20Publication%5D%20%3A%20%222013%22%5BDate%20-%20Publication%5D)" 
url = domain + url_tail 
 
browser = webdriver.Firefox() 
browser.get(url) 
time.sleep(5) 
 
def extract_data(browser): 
    titles = browser.find_elements_by_css_selector("div.rprt div.rslt p.title a") 
    return [title.text for title in titles] 
 
page_start = 1 
page_end = 10 
 
f = open('titles.txt', 'a') 
for page in range(page_start, page_end): 
    print "page %d" % page 
    page_jump_box = browser.find_element_by_class_name("num").clear() 
    page_jump_box_cleared = browser.find_element_by_class_name("num") 
    page_jump_box_cleared.send_keys(str(page) + Keys.RETURN) 
 
    time.sleep(15) 
 
    f = open('titles.txt', 'a') 
    for line in extract_data(browser): 
        f.write(line + '\n') 
 
f.close() 

当我运行它时,我得到了这个:

gao@gao:~/crawler$ python crawler3.0.py  
page 1 
page 2 
page 3 
page 4 
Traceback (most recent call last): 
  File "crawler3.0.py", line 33, in <module> 
    f.write(line + '\n') 
UnicodeEncodeError: 'ascii' codec can't encode character u'\u03b1' in position 36: ordinal not in range(128) 

当我在 Stackoverflow 上搜索时,我发现了一个类似的问题:UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 20: ordinal not in range(128) . 我知道当你使用 str() 时,它会导致 unicode 问题。但是在我的代码中,我只使用 str() 使 page 数字成为一个字符串。那么,如何更正代码。

这是另一个问题。我了解到,如果我想将 phantomjs 与 selenium 一起使用,我只需要将 browser = webdriver.Firefox() 更改为 browser = webdriver .PhantomJS(),但是当我这样做时,我抓取的内容被重复了(只抓取了第 1 页的标题)。

我的母语不是英语,如果有任何语法错误或任何错误,请告诉我。

提前致谢。

请您参考如下方法:

在写入文件之前,您需要对行进行编码:

for line in extract_data(browser): 
    f.write(line.encode('utf-8') + '\n') 

关于您的第二个问题,我建议进行以下改进(这将使其有效):

  • 使用Explicit Waits而不是 time.sleep() 调用 - 这也会显着提高性能
  • 不要键入页码,而是单击“下一步”按钮
  • 在循环之前以“追加”模式打开文件并使用with context manager
  • close() 完成后关闭浏览器

代码:

from selenium import webdriver 
from selenium.webdriver.common.by import By 
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC 
 
domain = "http://www.ncbi.nlm.nih.gov/" 
url_tail = "pubmed?term=(%222013%22%5BDate%20-%20Publication%5D%20%3A%20%222013%22%5BDate%20-%20Publication%5D)" 
url = domain + url_tail 
 
browser = webdriver.PhantomJS() 
browser.get(url) 
 
 
def extract_data(browser): 
    titles = browser.find_elements_by_css_selector("div.rprt div.rslt p.title a") 
    return [title.text for title in titles] 
 
 
page_start, page_end = 1, 10 
 
with open('titles.txt', 'a') as f: 
    for page in range(page_start, page_end): 
        WebDriverWait(browser, 10).until( 
            EC.presence_of_element_located((By.CSS_SELECTOR, "div.rprt p.title")) 
        ) 
 
        for line in extract_data(browser): 
            f.write(line.encode('utf-8') + '\n') 
 
        print "page %d" % page 
 
        browser.find_element_by_css_selector("div.pagination a.next").click() 
 
browser.close() 

这会生成 titles.txt,其中包含结果页 1-9 中的标题:

Robotic-assisted tubal anastomosis with one-stitch technique. 
Effectiveness of pictorial health warnings on cigarette packs among Lebanese school and university students. 
... 
Importance and globalization status of good manufacturing practice (GMP) requirements for pharmaceutical excipients. 
Systemic review on drug related hospital admissions - A pubmed based search.