我是Python和airflow的新手。我想在pythonoperator任务中使用http请求创建一个dataproc集群。请参见以下代码:
def create_cluster():
API_ENDPOINT = "https://dataproc.googleapis.com/v1beta2/projects/trim
-**********/regions/us-central1-b/clusters"
data = {
"projectId": "trim-**********",
"clusterName": "cluster-1",
"config": {
"configBucket": "",
"gceClusterConfig": {
"subnetworkUri": "default",
"zoneUri": "us-central1-b"
},
"masterConfig": {
"numInstances": 1,
"machineTypeUri": "n1-standard-1",
"diskConfig": {
"bootDiskSizeGb": 500,
"numLocalSsds": 0
}
},
"workerConfig": {
"numInstances": 2,
"machineTypeUri": "n1-standard-1",
"diskConfig": {
"bootDiskSizeGb": 100,
"numLocalSsds": 0
}
}
}
}
r = requests.post(url=API_ENDPOINT, data=data)
pastebin_url = r.text
print("The pastebin URL is:%s" % pastebin_url)
但是我遇到一个错误:
Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential.参见
https://developers.google.com/identity/sign-in/web/devconsole-project。
该错误的解决方案是什么?提前致谢。
请您参考如下方法:
代替使用PythonOperator创建集群,而是尝试使用专用的气流运算符,这里是一些documentation。
我认为使用DataprocClusterCreateOperator会更容易。
