Skip to main content
 首页 » 编程设计

python之Selenium Python之访问搜索结果的下一页

2024年11月01日11zhoujg

我必须从这个 url 中一个一个地点击每个搜索结果:

Search Guidelines

我首先从显示的文本中提取结果总数,以便我可以设置迭代上限

upperlimit=driver.find_element_by_id("total_results") 
number = int(upperlimit.text.split(' ')[0]) 

循环定义为 对于范围内的我(1,数字):

但是,在浏览第一页的前 10 个结果后,列表索引超出范围(可能是因为没有更多链接可供点击)。我需要单击“下一步”以获取接下来的 10 个结果,依此类推,直到完成所有搜索结果。我怎样才能做到这一点?

如有任何帮助,我们将不胜感激!

请您参考如下方法:

问题是id为total_results的元素的值在页面加载后发生了变化,一开始是117,然后变成了44.

相反,这里有一个更稳健的方法。它逐页处理,直到没有更多页面为止:

from selenium import webdriver 
from selenium.common.exceptions import NoSuchElementException 
 
driver = webdriver.Firefox() 
url = 'http://www.nice.org.uk/Search.do?searchText=bevacizumab&newsearch=true#/search/?searchText=bevacizumab&mode=&staticTitle=false&SEARCHTYPE_all2=true&SEARCHTYPE_all1=&SEARCHTYPE=GUIDANCE&TOPICLVL0_all2=true&TOPICLVL0_all1=&HIDEFILTER=TOPICLVL1&HIDEFILTER=TOPICLVL2&TREATMENTS_all2=true&TREATMENTS_all1=&GUIDANCETYPE_all2=true&GUIDANCETYPE_all1=&STATUS_all2=true&STATUS_all1=&HIDEFILTER=EGAPREFERENCE&HIDEFILTER=TOPICLVL3&DATEFILTER_ALL=ALL&DATEFILTER_PREV=ALL&custom_date_from=&custom_date_to=11-06-2014&PAGINATIONURL=%2FSearch.do%3FsearchText%40%40bevacizumab%26newsearch%40%40true%26page%40%40&SORTORDER=BESTMATCH' 
driver.get(url) 
 
page_number = 1 
while True: 
    try: 
        link = driver.find_element_by_link_text(str(page_number)) 
    except NoSuchElementException: 
        break 
    link.click() 
    print driver.current_url 
    page_number += 1 

基本上,这里的想法是获取下一页链接,直到没有这样的链接(将抛出 NoSuchElementException)。请注意,它适用于任意数量的页面和结果。

它打印:

http://www.nice.org.uk/Search.do?searchText=bevacizumab&newsearch=true&page=1 
http://www.nice.org.uk/Search.do?searchText=bevacizumab&newsearch=true&page=2#showfilter 
http://www.nice.org.uk/Search.do?searchText=bevacizumab&newsearch=true&page=3#showfilter 
http://www.nice.org.uk/Search.do?searchText=bevacizumab&newsearch=true&page=4#showfilter 
http://www.nice.org.uk/Search.do?searchText=bevacizumab&newsearch=true&page=5#showfilter