我在将新 CSV 文件写入 S3 存储桶时遇到问题。我希望能够读取 S3 存储桶中的 CSV 文件,如果 CSV 中的某个值符合特定要求,我想将其更改为不同的值。我读到过无法编辑 S3 对象,所以我每次都需要创建一个新对象。简而言之,我想从 S3 存储桶中的另一个 CSV 文件创建一个新的、更新的 CSV 文件,并应用更改。
我正在尝试使用 DictWriter 和 DictReader,但我总是遇到 DictWriter 的问题。我可以正确读取 CSV 文件,但是当我尝试更新它时,会出现无数与 DictWriter 截然不同的问题。现在,我遇到的问题是
# Function to be pasted into AWS Lambda.
# Accesses S3 bucket, opens the CSV file, receive the response line-by-line,
# To be able to access S3 buckets and the objects within the bucket
import boto3
# To be able to read the CSV by using DictReader
import csv
# Lambda script that extracts, transforms, and loads data from S3 bucket 'testing-bucket-1042' and CSV file 'Insurance.csv'
def lambda_handler(event, context):
s3 = boto3.resource('s3')
bucket = s3.Bucket('testing-bucket-1042')
obj = bucket.Object(key = 'Insurance.csv')
response = obj.get()
lines = response['Body'].read().decode('utf-8').split()
reader = csv.DictReader(lines)
with open("s3://testing-bucket-1042/Insurance.csv", newline = '') as csvfile:
reader = csv.DictReader(csvfile)
fieldnames = ['county', 'eq_site_limit']
writer = csv.DictWriter(lines, fieldnames=fieldnames)
for row in reader:
writer.writeheader()
if row['county'] == "CLAY": # if the row is under the column 'county', and contains the string "CLAY"
writer.writerow({'county': 'CHANGED'})
if row['eq_site_limit'] == "0": # if the row is under the column 'eq_site_limit', and contains the string "0"
writer.writerow({'eq_site_limit': '9000'})
现在,我遇到的错误是我在尝试打开 CSV 文件时使用的路径“s3://testing-bucket-1042/Insurance.csv”据说不存在。
错误说
"errorMessage": "[Errno 2] 没有这样的文件或目录:'s3://testing-bucket-1042/Insurance.csv'", “errorType”:“FileNotFoundError”
如果有的话,使用 DictWriter 的正确方法是什么?
请您参考如下方法:
首先 s3:\\ 不是一个通用的(文件)协议(protocol),因此您会收到错误消息。很好,你表明了你的意图。
好的,我重构了你的代码
import codecs
import boto3
# To be able to read the CSV by using DictReader
import csv
from io import StringIO
# Lambda script that extracts, transforms, and loads data from S3 bucket 'testing-bucket-1042' and CSV file 'Insurance.csv'
def lambda_handler(event, context):
s3 = boto3.resource('s3')
bucket = s3.Bucket('testing-bucket-1042')
obj = bucket.Object(key = 'Insurance.csv')
stream = codecs.getreader('utf-8')(obj.get()['Body'])
lines = list(csv.DictReader(stream))
### now you have your object there
csv_buffer = StringIO()
out = csv.DictWriter(csv_buffer, fieldnames=['county', 'eq_site_limit'])
for row in lines:
if row['county'] == "CLAY":
out.writerow({'county': 'CHANGED'})
if row['eq_site_limit'] == "0":
out.writerow({'eq_site_limit': '9000'})
### now write content into some different bucket/key
s3client = boto3.client('s3')
s3client.put_object(Body=csv_buffer.getvalue().encode(encoding),
Bucket=...targetbucket, Key=...targetkey)
我希望这能奏效。基本上有几个技巧:
- 使用
codecs直接从s3 bucket流式传输csv数据 - 使用
BytesIO在内存中创建一个流,csv.DictWriter可以写入该流。 - 完成后,“上传”内容的一种方法是通过
s3.clients的put_object方法(如 AWS 中所述)
