Skip to main content
 首页 » 编程设计

python之如果表 beautifulsoup 和 pandas 中不存在类,则停止抓取 url

2024年12月31日8tuyile006

我正在使用 csv 文件中的 url 列表从 html 表中抓取和提取数据。当表中不存在“style3”时,我想停止浏览 url。 我创建了一个函数,如果它不存在,它将返回 false,但我对如何实际实现它感到困惑。

任何关于解决方案的建议或对文献的指导都会有很大帮助,因为我在这里找不到任何可以帮助我弄清楚的东西。

我添加了 1 个带有“style3”的 url 和 1 个不带“style3”的 url。感谢您提供的所有帮助。

http://www.wvlabor.com/new_searches/contractor_RESULTS.cfm?wvnumber=WV057808&contractor_name=&dba=&city_name=&County=&Submit3=Search+Contractors http://www.wvlabor.com/new_searches/contractor_RESULTS.cfm?wvnumber=WV057924&contractor_name=&dba=&city_name=&County=&Submit3=Search+Contractors

import csv 
from urllib.request import urlopen 
import pandas as pd 
from bs4 import BeautifulSoup as BS 
 
def license_exists(soup): 
    contents = [] 
    with open('WV_urls.csv','r') as csvf: 
        urls = csv.reader(csvf) 
        for url in urls: 
            if soup(class_='style3'): 
                return True 
            else: 
                return False 
 
contents = [] 
more = True 
while more: 
    df  = pd.DataFrame(columns=['WV Number', 'Company', 'DBA', 'Address', 'City', 'State', 'Zip','County', 'Phone', 'Classification*', 'Expires']) #initialize the data frame with columns 
    with open('WV_urls.csv','r') as csvf: # Open file in read mode 
        urls = csv.reader(csvf) 
        for url in urls: 
            contents.append(url) # Add each url to list contents 
        for url in contents:  # Parse through each url in the list. 
            page = urlopen(url[0]).read() 
            df1, header = pd.read_html(page,header=0)#reading with header 
            more = license_exists(?????) 
            df=df.append(df1) # append to dataframe 
 
            df.to_csv('WV_Licenses_Daily.csv', index=False) 

请您参考如下方法:

您可以使用单个 for 循环和中断来完成此操作(不需要 while more):

lst = [] 
with open('WV_urls.csv','r') as csvf: # Open file in read mode 
    urls = csv.reader(csvf) 
    for url in urls: 
        page = urlopen(url[0]).read() 
        df1, header = pd.read_html(page, header=0) 
        if license_exists(BS(page, ‘html.parser’)): 
            # if the license is present we don't want to parse any more urls. 
            # Note: we don't append this last result (should we?) 
            break 
        lst.append(df1) 
 
df = pd.concat(lst) 
df.to_csv('WV_Licenses_Daily.csv', index=False) 

注意:这会从 DataFrame 列表中创建最终的 DataFrame,这比每次附加都更有效。