Skip to main content
 首页 » 编程设计

python之如何在python中从网络抓取的数据写入csv文件

2024年12月31日8lyj

我正在尝试从网页中抓取数据并且也能够抓取。 使用下面的脚本获取所有 div 类数据后,但我很困惑如何在 CSV 文件中写入数据。

名字列中的第一个数据 姓氏列中的姓氏数据 . .

from urllib.request import urlopen 
from bs4 import BeautifulSoup 
 
html = 'http://rerait.telangana.gov.in/PrintPreview/PrintPreview/UHJvamVjdElEPTQmRGl2aXNpb249MSZVc2VySUQ9MjAyODcmUm9sZUlEPTEmQXBwSUQ9NSZBY3Rpb249U0VBUkNIJkNoYXJhY3RlckQ9MjImRXh0QXBwSUQ9' 
 
page = urlopen(html) 
 
data = BeautifulSoup(page, 'html.parser') 
 
name_box = data.findAll('div', attrs={'class': 'col-md-3 col-sm-3'}) #edited companyName_99a4824b -> companyName__99a4824b 
 
for i in range(len(name_box)): 
    data = name_box[i].text.strip() 

数据:

Information Type 
Individual 
 
First Name 
KACHAM 
Middle Name 
 
Last Name 
RAJESHWAR 
Father Full Name 
RAMAIAH 
Do you have any Past Experience ? 
No 
Do you have any registration in other State than registred State? 
No 
House Number 
8-2-293/82/A/446/1 
Building Name 
SAI KRUPA 
Street  Name 
ROAD NO 20 
Locality 
JUBILEE HILLS 
Landmark 
JUBILEE HILLS 
State 
Telangana 
Division 
Division 1 
District 
Hyderabad 
Mandal 
Shaikpet 
Village/City/Town 
 
Pin Code 
500033 
Office Number 
04040151614 
Fax Number 
 
Website URL 
 
Authority Name 
 
Plan Approval Number 
1/18B/06558/2018 
Project Name 
SKV S ANANDA VILAS 
Project Status 
New Project 
Proposed Date of Completion 
17/04/2024 
Litigations related to the project ? 
No 
Project Type 
Residential 
Are there any Promoter(Land Owner/ Investor) (as defined by Telangana RERA Order) in the project ? 
Yes 
Sy.No/TS No. 
00 
Plot No./House No. 
10-2-327 
Total Area(In sqmts) 
526.74 
Area affected in Road widening/FTL of Tanks/Nala Widening(In sqmts) 
58.51 
Net Area(In sqmts) 
1 
Total Building Units (as per approved plan) 
1 
Proposed Building Units(as per agreement) 
1 
 
 
Boundaries East 
PLOT NO 213 
Boundaries West 
PLOT NO 215 
Boundaries North 
PLOT NO 199 
Boundaries South 
ROAD NO 8 
Approved Built up Area (In Sqmts) 
1313.55 
Mortgage Area  (In Sqmts) 
144.28 
State 
Telangana 
District 
Hyderabad 
Mandal 
Maredpally 
Village/City/Town 
 
Street 
ROAD NO 8 
Locality 
SECUNDERABAD COURT 
Pin Code 
500026 

以上是运行上述代码后得到的数据。

编辑

for i in range(len(name_box)): 
    data = name_box[i].text.strip() 
    print (data) 
    fname = 'out.csv' 
    with open(fname) as f: 
        next(f) 
        for line in f: 
            head = [] 
            value = [] 
            for row in line: 
                head.append(row) 
            print (row) 

预期

Information Type | First  | Middle Name | Last Name | ...... 
Individual       | KACHAM |             | RAJESHWAR | ..... 

我有 200 个 url,但所有 url 数据都不相同,这意味着其中一些丢失了。如果数据不可用,我想这样写,然后只写空白。

请推荐。提前谢谢你

请您参考如下方法:

要写入 csv,您需要知道 head 和 body 中应该有什么值,在这种情况下,head 值应该是 html 元素包含 <label

from urllib2 import urlopen 
from bs4 import BeautifulSoup 
 
html = 'http://rerait.telangana.gov.in/PrintPreview/PrintPreview/UHJvamVjdElEPTQmRGl2aXNpb249MSZVc2VySUQ9MjAyODcmUm9sZUlEPTEmQXBwSUQ9NSZBY3Rpb249U0VBUkNIJkNoYXJhY3RlckQ9MjImRXh0QXBwSUQ9' 
 
page = urlopen(html) 
 
data = BeautifulSoup(page, 'html.parser') 
 
name_box = data.findAll('div', attrs={'class': 'col-md-3 col-sm-3'}) #edited companyName_99a4824b -> companyName__99a4824b 
 
heads = [] 
values = [] 
 
for i in range(len(name_box)): 
    data = name_box[i].text.strip() 
    dataHTML = str(name_box[i]) 
    if 'PInfoType' in dataHTML: 
        # <div class="col-md-3 col-sm-3" id="PInfoType"> 
        # empty value, maybe additional data for "Information Type" 
        continue 
 
    if 'for="2"' in dataHTML: 
        # <label for="2">No</label> 
        # it should be head but actually value 
        values.append(data) 
 
    elif '<label' in dataHTML: 
        # <label for="PersonalInfoModel_InfoTypeValue">Information Type</label> 
        # head or top row 
        heads.append(data) 
 
    else: 
        # <div class="col-md-3 col-sm-3">Individual</div> 
        # value for second row 
        values.append(data) 
 
csvData = ', '.join(heads) + '\n' + ', '.join(values)     
with open("results.csv", 'w') as f: 
    f.write(csvData) 
 
print "finish."