Skip to main content
 首页 » 编程设计

python之实时网页抓取

2024年12月31日9kerrycode

我目前正在使用 BeautifulSoup 进行网络抓取,它是在 xml 中获取和编写的,如下面的代码所示,我只是想知道我该怎么做才能使其实时,因为网站每 5 分钟更新一次。

import csv 
import requests 
from bs4 import BeautifulSoup 
 
url = 'http://www.dublincity.ie/dublintraffic/cpdata.xml?1543254514266' 
 
res = requests.get(url) 
soup = BeautifulSoup(res.content,"xml") 
data = [] 
for item in soup.select("carpark"): 
    ditem = {} 
    ditem['Name'] = item.get("name") 
    ditem['Spaces'] = item.get("spaces") 
    data.append(ditem) 
 
with open("xmldocs.csv","w",newline="") as f: 
    writer = csv.DictWriter(f,["Name","Spaces"]) 
    writer.writeheader() 
    for info in data: 
        writer.writerow(info) 

请您参考如下方法:

您可以使用 while循环,然后在最后你可以添加一个 sleep 5 分钟。

使用您的示例,这将是:

import csv 
import requests 
from bs4 import BeautifulSoup 
import time 
 
while True: 
    url = 'http://www.dublincity.ie/dublintraffic/cpdata.xml?1543254514266' 
 
    res = requests.get(url) 
    soup = BeautifulSoup(res.content,"xml") 
    data = [] 
    for item in soup.select("carpark"): 
        ditem = {} 
        ditem['Name'] = item.get("name") 
        ditem['Spaces'] = item.get("spaces") 
        data.append(ditem) 
 
    with open("xmldocs.csv","w",newline="") as f: 
        writer = csv.DictWriter(f,["Name","Spaces"]) 
        writer.writeheader() 
        for info in data: 
            writer.writerow(info) 
 
    time.sleep(5 * 60)