Skip to main content
 首页 » 编程设计

python之在网页上应用过滤器时如何复制请求

2024年11月01日8kerrycode

我正在尝试在 premier league player stats 中应用过滤器时复制请求.我注意到 url 在为 2019/20 赛季应用过滤器时添加了组件“?co=1&se=274”

https://www.premierleague.com//players/5140/Virgil-van-Dijk/stats?co=1&se=274 

代替

https://www.premierleague.com//players/5140/Virgil-van-Dijk/stats 

但是在做的时候

requests.get('https://www.premierleague.com//players/5140/Virgil-van-Dijk/stats?co=1&se=274') 

并抓取内容,它被抓取就像没有应用过滤器一样。如何在网页上应用过滤器的地方发出请求?

通过深入研究,我了解到它受 CloudFront 保护,这意味着在发布请求之前所有查询参数都会被删除。有解决办法吗?

这是我抓取数据的方式:

from bs4 import BeautifulSoup as soup 
import requests 
from tqdm import tqdm 
from pprint import pprint 
players_url =['https://www.premierleague.com//players/5140/Virgil-van-Dijk/stats?co=1&se=274'] 
 
 
# this is dict where we store all information: 
players = {} 
for i in tqdm(players_url): 
    player_page = requests.get(i) 
    cont = soup(player_page.content, 'lxml') 
    time.sleep(2) 
    data = dict((k.contents[0].strip(), v.get_text(strip=True)) for k, v in zip(cont.select('.topStat span.stat, .normalStat span.stat'), cont.select('.topStat span.stat > span, .normalStat span.stat > span'))) 
    clud_ele = cont.find('div', attrs={'class' : 'info'}) 
    club = {"Club" : clud_ele.get_text(strip=True)} 
    position = {"Position": clud_ele.find_next('div', attrs={'class' : 'info'}).get_text(strip=True)} 
    data.update(club) 
    data.update(position) 
    players[cont.select_one('.playerDetails .name').get_text(strip=True)] = data 
 
pprint(players) 

在输出中我可以清楚地看到没有应用过滤器,因为本赛季还没有 45 场比赛

{'Virgil van Dijk': {'Accurate long balls': '533', 
                     'Aerial battles lost': '207', 
                     'Aerial battles won': '589', 
                     'Appearances': '122', 
                     'Assists': '2', 
                     'Big chances created': '11', 
                     'Blocked shots': '23', 
                     'Clean sheets': '45', 

请您参考如下方法:

您可以通过复制当您尝试按季节过滤时完成的后台请求来解决这个问题。我使用 requests 库来获取所有玩家的统计数据。

这个过程主要涉及三个url,

  1. 获取季节 ID 值。 (例如 274)

https://footballapi.pulselive.com/football/competitions/1/compseasons?page=0&pageSize=100

  1. 获取球员姓名和 ids (例如姓名:Virgil van Dijk,ID:5140)

https://footballapi.pulselive.com/football/players

  1. 使用玩家 ID (例如 5140)获取玩家统计数据

https://footballapi.pulselive.com/football/stats/player/

完整脚本

import requests 
import json 
 
response = requests.get('https://footballapi.pulselive.com/football/competitions/1/compseasons?page=0&pageSize=100').json() # request to obtain the id values and corresponding season  
 
id = int(response["content"][0]["id"]) # converts current season id which is a decimal point value to interger 
 
players = {} # dictionary to store players data 
 
playersAndStats = {} # dictionary to store player name and associated stats 
 
numEntries = 100 
 
page = 0 
 
# loop to get player name and id 
while True:  
 
  params = ( 
      ('pageSize', '100'), 
      ('compSeasons', str(id)),  
      ('altIds', 'true'), 
      ('page', str(page)), 
      ('type', 'player'), 
      ('id', '-1'), 
      ('compSeasonId', str(id)), 
  ) 
 
  response = requests.get('https://footballapi.pulselive.com/football/players',params=params).json() 
 
  playersData = response["content"] 
 
  for playerData in playersData: 
    players[playerData["id"]] = playerData["name"]["display"] 
 
  page += 1 
 
  if page == response["pageInfo"]["numPages"]: 
    break 
 
print("Total no. of players :",len(players)) 
 
count = 0  
total = len(players) 
 
# loop to get player stats  
for player in players: 
 
  count += 1 
  print(count,"/",total) 
 
  params = ( 
      ('comps', '1'), 
      ('compSeasons', str(id)), # setting season id to current season id 
  ) 
 
  playerId = str(int(player)) 
 
  # gets the stat of the player using playerId  
  response = requests.get('https://footballapi.pulselive.com/football/stats/player/'+playerId,params=params).json() 
 
  stats = response["stats"] 
 
  # creating a stat dict for the player 
  playersAndStats[players[player]] = {} 
 
  # loop to store each stat associated with the player 
  for stat in stats: 
    playersAndStats[players[player]][stat["name"].replace("_"," ").capitalize()] = int(stat["value"]) 
 
# to store data to a json file  
f = open("data.json","w") 
 
# pretty prints and writes the same to the json file  
f.write(json.dumps(playersAndStats,indent=4, sort_keys=True)) 
f.close() 
 
print("Saved to data.json") 

示例输出

"Aaron Connolly": { 
        "Accurate back zone pass": 1, 
        "Accurate fwd zone pass": 1, 
        "Accurate pass": 2, 
        "Aerial lost": 1, 
        "Appearances": 1, 
        "Attempts conceded ibox": 2, 
        "Attempts conceded obox": 1, 
        "Backward pass": 1, 
        "Ball recovery": 1, 
        "Duel lost": 1, 
        "Duel won": 0, 
        "Final third entries": 1, 
        "Goals conceded": 1, 
        "Goals conceded ibox": 1, 
        "Losses": 1, 
        "Mins played": 24, 
        "Open play pass": 1, 
        "Poss won mid 3rd": 1, 
        "Rightside pass": 1, 
        "Successful final third passes": 1, 
        "Successful open play pass": 1, 
        "Total back zone pass": 1, 
        "Total final third passes": 1, 
        "Total fwd zone pass": 1, 
        "Total offside": 2, 
        "Total pass": 2, 
        "Total sub on": 1, 
        "Total tackle": 0, 
        "Touches": 3, 
        "Won tackle": 0 
    } 

data.json 文件包含所有玩家的数据。

The stats associated with each player varies according to their playing position.