Platform:
Python version: 3.7.3
Selenium Version: 3.141.0
OS: Win7
问题:
我有一个作为文本文件的 url 列表,每个 url 单独一行。网址是下载链接。我想遍历所有网址并将链接到每个网址的文件下载到特定文件夹中。
我试过的代码是一个嵌套的 for-while 循环。 第一次迭代没有任何问题,但第二次迭代卡在了一个 while 循环中。
显然有更好的方法来完成我想做的事情。我只是 Python 的初学者,正在尽我所能学习这门语言。
My Url List:
https://mega.nz/#!bOgBWKiB!AWs3JSksW0mpZ8Eob0-Qpr5ZAG0N1zhoFBFVstNJfXs
https://mega.nz/#!qPxGAAYJ!BX-hv7jgE4qvBs_uhHPVpsLRm1Yl4JkZ17nI1-U6hvk
https://mega.nz/#!GPoiHaaT!TAKT4sOhIiMUSFFSmlvPOidMcscXzHH_8HgK27LyTRM
我试过的代码:
import os
from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep
from pathlib import Path
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
binary = FirefoxBinary('C:\\Program Files\\Mozilla Firefox\\firefox.exe')
fp = webdriver.FirefoxProfile()
fp.set_preference("browser.download.folderList", 2)
fp.set_preference("browser.download.manager.showWhenStarting", False)
fp.set_preference("browser.download.dir", "H:\\downloads")
fp.set_preference("browser.helperApps.neverAsk.saveToDisk", "application/zip")
driver = webdriver.Firefox(firefox_binary=binary, firefox_profile=fp, executable_path=r'C:\\Program Files\\Python\\Python37\\Lib\\site-packages\\selenium\\webdriver\\firefox\\geckodriver.exe')
driver.set_window_size(1600, 1050)
with open("H:\\downloads\\my_url_list.txt", "r") as f:
for url in f:
driver.get(url.strip())
sleep(5)
while True:
# checks whether the element is available on the page, used 'while' instead of 'wait' as I couuldn't figure out the wait time.
try:
content = driver.find_element_by_css_selector('div.buttons-block:nth-child(1) > div:nth-child(2)')
break
except NoSuchElementException:
continue
# used 'execute_script' instead of 'click()' due to "scroll into view error"
driver.execute_script("arguments[0].click();", content)
sleep(5)
while True:
# checks whether 'filename' element is available on the page, the page shows multiple elements depending on interaction.
if driver.find_element_by_xpath("/html/body/div[6]/div[3]/div/div[1]/div[4]/div[1]/div/span[1]"):
filename = driver.find_element_by_xpath("/html/body/div[6]/div[3]/div/div[1]/div[4]/div[1]/div/span[1]").text
break
elif driver.find_element_by_xpath("/html/body/div[6]/div[3]/div/div[1]/div[5]/div/div/div[1]/div[1]/div[2]/div[3]/div[1]/span[1]"):
filename = driver.find_element_by_xpath("/html/body/div[6]/div[3]/div/div[1]/div[5]/div/div/div[1]/div[1]/div[2]/div[3]/div[1]/span[1]").text
break
else:
sleep(5)
print(filename)
dirname = 'H:\\downloads'
suffix = '.zip'
file_path = Path(dirname, filename).with_suffix(suffix)
while True:
# checks whether the file has downloaded into the folder.
if os.path.isfile(file_path):
break
发生了什么:
第一次迭代 - 文件(链接到 url)被下载到 H:\\downloads 文件夹中,filename 被打印出来。
在第二次迭代的情况下,文件被下载到文件夹中但文件名没有被打印,涉及的第二个 while 循环进入无限循环。
第二次运行后没有for迭代,因为在第二次迭代中无法检索filename,循环进入无限模式。
上面代码中的第二个 while 循环:
while True:
# checks whether 'filename' element is available on the page, the page shows multiple elements depending on interaction.
if driver.find_element_by_xpath("/html/body/div[6]/div[3]/div/div[1]/div[4]/div[1]/div/span[1]"):
filename = driver.find_element_by_xpath("/html/body/div[6]/div[3]/div/div[1]/div[4]/div[1]/div/span[1]").text
break
elif driver.find_element_by_xpath("/html/body/div[6]/div[3]/div/div[1]/div[5]/div/div/div[1]/div[1]/div[2]/div[3]/div[1]/span[1]"):
filename = driver.find_element_by_xpath("/html/body/div[6]/div[3]/div/div[1]/div[5]/div/div/div[1]/div[1]/div[2]/div[3]/div[1]/span[1]").text
break
else:
sleep(5)
附上文件名xpath选项的图片(文件名选择两个不同xpath的原因)
- while循环第一个选项
- while 循环第二个选项
请您参考如下方法:
你搜索的是显式等待,建议你访问这个page来自 Selenium-python 文档。 我从页面引用:
An explicit wait is a code you define to wait for a certain condition to occur before proceeding further in the code. The extreme case of this is time.sleep(), which sets the condition to an exact time period to wait. There are some convenience methods provided that help you write code that will wait only as long as required. WebDriverWait in combination with ExpectedCondition is one way this can be accomplished.
如果您想了解有关 ExpectedCondition 的更多信息,可以访问此 link文档的
我建议您使用 lambda 函数为您的案例编写这段代码,因为您至少在等待一个元素。
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox()
driver.get("http://somedomain/url_that_delays_loading")
try:
xpath1="/html/body/div[6]/div[3]/div/div[1]/div[4]/div[1]/div/span[1]"
xpath2="/html/body/div[6]/div[3]/div/div[1]/div[5]/div/div/div[1]/div[1]/div[2]/div[3]/div[1]/span[1]"
timeLimit = 15 #seconds, you really need to set a time out.
element = WebDriverWait(driver, timeLimit).until( lambda driver: driver.find_elements(By.xpath, xpath1) or driver.find_elements(By.xpath, xpath2) )
finally:
pass
这会在抛出 TimeoutException 之前最多等待 15 秒,除非它通过 xpath 找到您正在等待的元素之一。默认情况下,WebDriverWait 每 500 毫秒调用一次 ExpectedCondition,直到它成功返回,因此您不需要像您尝试的那样处理逻辑和循环。
例如,为了处理 TimeoutException,您可以刷新页面。
