Skip to main content
 首页 » 编程设计

python之无法使用请求从相同的网址收集个人资料名称

2024年11月01日11langtianya

我正在尝试使用 selenium 登录 facebook,然后将 cookies 传输到 requests 模块,以便我可以从中收集个人资料名称使用 requests 的两个 url。两个网址中可用的配置文件名称不是动态的,但它们确实需要登录。

我的以下脚本可以成功登录,但在传输 cookies 时可能出现问题,这可能是脚本在点击此行 时抛出 AttributeError 的原因name = soup.select_one("#fb-timeline-cover-name > a").text.

到目前为止我已经写了:

import requests 
from bs4 import BeautifulSoup 
from selenium import webdriver 
from selenium.webdriver.common.keys import Keys 
 
url = "https://www.facebook.com/" 
 
links = [ 
    "https://www.facebook.com/hillsendagain?fref=gm&dti=157300781073597&hc_location=group", 
    "https://www.facebook.com/mark.porton.9?fref=gm&dti=157300781073597&hc_location=group" 
] 
 
chrome_options = webdriver.ChromeOptions() 
prefs = {"profile.default_content_setting_values.notifications" : 2} 
chrome_options.add_experimental_option("prefs",prefs) 
driver = webdriver.Chrome(options=chrome_options) 
 
driver.get(url) 
driver.find_element_by_id("email").send_keys("username") 
driver.find_element_by_id("pass").send_keys("password",Keys.RETURN) 
 
driver_cookies = driver.get_cookies() 
c = {c['name']:c['value'] for c in driver_cookies} 
 
for link in links: 
    res = requests.get(link,headers={'User-Agent':'Mozilla/5.0'},cookies=c) 
    soup = BeautifulSoup(res.text,"lxml") 
    name = soup.select_one("#fb-timeline-cover-name > a").text 
    print(name) 
driver.quit() 

如何使用请求仅获取配置文件名称?

PS Getting profile names using selenium alone is not I'm after coz I already know how to do that.

请您参考如下方法:

虽然你感兴趣的内容不是动态的,但是都被注释掉了。尝试以下方法来实现:

for link in links: 
    content = requests.get(link,headers={'User-Agent':'Mozilla/5.0'},cookies=c).text 
    comment = content.replace("-->", "").replace("<!--", "") 
    soup = BeautifulSoup(comment,"lxml") 
    name = soup.select_one("#fb-timeline-cover-name > a").text 
    print(name) 

在我看来,使用 session 是你想要让它变得健壮的方法:

s = requests.Session() 
[s.cookies.set(cookie['name'],cookie['value']) for cookie in driver.get_cookies()] 
 
for link in links: 
    content = s.get(link,headers={'User-Agent':'Mozilla/5.0'}).text 
    comment = content.replace("-->", "").replace("<!--", "") 
    soup = BeautifulSoup(comment,"lxml") 
    name = soup.select_one("#fb-timeline-cover-name > a").text 
    print(name)