Skip to main content
 首页 » 编程设计

python之需要接受隐私政策才能访问页面

2024年12月31日7dflying

我正在尝试从 AllRecipes.co.uk 抓取信息,但是当代码运行时,我没有被定向到预期的页面,而是被定向到一个要求我事先接受隐私政策的封面。这意味着我无法从我想要的页面上抓取,因为我访问的任何页面都带有此接受隐私政策封面

网站是 AllRecipes.co.uk

import requests 
from bs4 import BeautifulSoup 
import time 
from selenium import webdriver 
import numpy as np 
import os 
 
 
userAgent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36' 
headers = { 
        'user-agent': userAgent 
    } 
 
dishType = "main-recipes" 
url = 'http://allrecipes.co.uk/recipes/' + dishType + '.aspx?page=' 
#endPage = 1259 
endPage = 3 
for i in range(2, endPage): 
    delays = [5, 7, 9, 11, 13, 15] 
    delay = np.random.choice(delays) 
    time.sleep(delay) 
    print("Getting request " + str(i)) 
    r = requests.get(url + str(i)) 
    soup = BeautifulSoup(r.content, "html.parser") 
    print(soup) 
    #names = soup.findAll('div', attrs = {'class' : "col-sm-7"}) 
    #for name in names: 
    #    print(name) 

请您参考如下方法:

您只需要设置 euConsentId cookie:

In [1]: import requests 
 
In [2]: from bs4 import BeautifulSoup 
 
In [3]: url = "http://allrecipes.co.uk/recipes/main-recipes.aspx?page=2" 
 
In [4]: BeautifulSoup(requests.get(url).content, "html.parser").title.get_text() 
Out[4]: 'About your privacy on this site' 
 
In [5]: import uuid 
 
In [6]: BeautifulSoup(requests.get(url, cookies={'euConsentId': str(uuid.uuid4())}).content, "html.parser").title.get_text() 
Out[6]: 'Main course recipes - All recipes UK ' 

为了在您的代码中进行调整,我将实例化一个 "session"并在那里设置 cookie:

import uuid4 
 
consent_id = str(uuid.uuid4()) 
with requests.Session() as session: 
    session.cookies = {'euConsentId': consent_id} 
 
    response = session.get(...)