我正在尝试使用 selenium 登录 facebook,然后将 cookies 传输到 requests 模块,以便我可以从中收集个人资料名称使用 requests 的两个 url。两个网址中可用的配置文件名称不是动态的,但它们确实需要登录。
我的以下脚本可以成功登录,但在传输 cookies 时可能出现问题,这可能是脚本在点击此行 时抛出 .AttributeError 的原因name = soup.select_one("#fb-timeline-cover-name > a").text
到目前为止我已经写了:
import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
url = "https://www.facebook.com/"
links = [
"https://www.facebook.com/hillsendagain?fref=gm&dti=157300781073597&hc_location=group",
"https://www.facebook.com/mark.porton.9?fref=gm&dti=157300781073597&hc_location=group"
]
chrome_options = webdriver.ChromeOptions()
prefs = {"profile.default_content_setting_values.notifications" : 2}
chrome_options.add_experimental_option("prefs",prefs)
driver = webdriver.Chrome(options=chrome_options)
driver.get(url)
driver.find_element_by_id("email").send_keys("username")
driver.find_element_by_id("pass").send_keys("password",Keys.RETURN)
driver_cookies = driver.get_cookies()
c = {c['name']:c['value'] for c in driver_cookies}
for link in links:
res = requests.get(link,headers={'User-Agent':'Mozilla/5.0'},cookies=c)
soup = BeautifulSoup(res.text,"lxml")
name = soup.select_one("#fb-timeline-cover-name > a").text
print(name)
driver.quit()
如何使用请求仅获取配置文件名称?
PS Getting profile names using selenium alone is not I'm after coz I already know how to do that.
请您参考如下方法:
虽然你感兴趣的内容不是动态的,但是都被注释掉了。尝试以下方法来实现:
for link in links:
content = requests.get(link,headers={'User-Agent':'Mozilla/5.0'},cookies=c).text
comment = content.replace("-->", "").replace("<!--", "")
soup = BeautifulSoup(comment,"lxml")
name = soup.select_one("#fb-timeline-cover-name > a").text
print(name)
在我看来,使用 session 是你想要让它变得健壮的方法:
s = requests.Session()
[s.cookies.set(cookie['name'],cookie['value']) for cookie in driver.get_cookies()]
for link in links:
content = s.get(link,headers={'User-Agent':'Mozilla/5.0'}).text
comment = content.replace("-->", "").replace("<!--", "")
soup = BeautifulSoup(comment,"lxml")
name = soup.select_one("#fb-timeline-cover-name > a").text
print(name)
