Skip to main content
 首页 » 编程设计

python之将抓取的数据保存到 csv 文件中

2025年02月15日14leader

我正在迭代一个过程,在该过程中,我将 Python 定向到一个网站,并指示 Python 查找我在指定网站的 csv 文件中的地址。我想告诉 Python 将网站上每个地址值的结果保存到一个 csv 文件中。

from selenium import webdriver 
from bs4 import BeautifulSoup 
import time 
import csv 
 
 
driver = webdriver.Chrome("C:\Python27\Scripts\chromedriver.exe") 
chrome = driver.get('https://etrakit.friscotexas.gov/Search/permit.aspx') 
with open('C:/Users/thefirstcolumnedited.csv','r') as f: 
    addresses = f.readlines() 
 
    for address in addresses: 
        driver.find_element_by_css_selector('#cplMain_txtSearchString').clear()        
        driver.find_element_by_css_selector('#cplMain_txtSearchString').send_keys(address) 
        driver.find_element_by_css_selector('#cplMain_btnSearch').click() 
        time.sleep(5) 
 
    soup = BeautifulSoup(chrome, 'html.parser') 
 
    writer = csv.writer(open('thematchingresults.csv', 'w')) 
    writer.writerow(soup) 

例如:

 6579 Mountain Sky Rd 

上面的地址值从网站中检索了五行数据。我如何告诉 Beautiful Soup 将每个地址值的结果保存在 csv 文件中?

请您参考如下方法:

想法是在循环内写入 CSV 文件(如果要为所有输入地址生成单个 csv 文件,请使用 a “追加”模式)。至于提取结果,我会 explicitly wait (time.sleep() 不可靠,通常比应有的速度慢)结果 table 元素(带有 id="ctl00_cplMain_rgSearchRslts_ctl00" 的元素>),然后使用 pandas.read_html()将 table 读入 dataframe,然后我们可以通过 .to_csv() 方便地转储到 CSV 文件中:

import pandas as pd 
from selenium import webdriver 
from selenium.webdriver.common.by import By 
from selenium.webdriver.support.wait import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC 
 
# ... 
 
wait = WebDriverWait(driver, 10) 
 
for address in addresses: 
    driver.find_element_by_css_selector('#cplMain_txtSearchString').clear() 
    driver.find_element_by_css_selector('#cplMain_txtSearchString').send_keys(address) 
    driver.find_element_by_css_selector('#cplMain_btnSearch').click() 
 
    # wait for the results table 
    table = wait.until(EC.visibility_of_element_located((By.ID, "ctl00_cplMain_rgSearchRslts_ctl00"))) 
 
    # make a dataframe and dump the results 
    df = pd.read_html(table.get_attribute("outerHTML"))[0] 
    with open('thematchingresults.csv', 'a') as f: 
        df.to_csv(f) 

对于单个“6579 Mountain Sky Rd”地址,运行脚本后 thematchingresults.csv 的内容将是:

,Permit Number,Address,Street Name,Applicant Name,Contractor Name,SITE_SUBDIVISION,RECORDID 
0,B13-2809,6579 MOUNTAIN SKY RD,MOUNTAIN SKY RD,SHADDOCK HOMES LTD,SHADDOCK HOMES LTD,PCR - SHERIDAN,MAC:1308050328358768 
1,B13-4096,6579 MOUNTAIN SKY RD,MOUNTAIN SKY RD,MIRAGE CUSTOM POOLS,MIRAGE CUSTOM POOLS,PCR - SHERIDAN,MAC:1312030307087756 
2,L14-1640,6579 MOUNTAIN SKY RD,MOUNTAIN SKY RD,TDS IRRIGATION,TDS IRRIGATION,SHERIDAN,ECON:140506012624706 
3,P14-0018,6579 MOUNTAIN SKY RD,MOUNTAIN SKY RD,MIRAGE CUSTOM POOLS,,SHERIDAN,LCR:1401130949212891 
4,ROW14-3205,6579 MOUNTAIN SKY RD,MOUNTAIN SKY RD,Housley Group,Housley Group,,TLW:1406190424422330 

希望这对您来说是一个好的起点。