AWS - key error partition_url - Witsml parser
Here is the whole process but it's likely an Airflow configuration issue :
Full report :
About to upload the following file : energistics_well.xml
Access/ID Token retrieval
Semi Manual flow from AWS
File upload
Get the SignedUrl from the Dataset API
GET https://r3.../getStorageInstructions?kindSubType=dataset--File.Generic
-
Given Authorization access/id token
-
Given data-partition-id : osdu
-
Unsigned Url : s3://osr3m10-sh... Need to submit metadata ? or register ! for dataset (using unsigned url)
File upload
PUT https://osr3m10...
- Given headers :
Content-Type: Application/xml
- Given Binary file stream
Upload file response code 200
File registration
Given body :
{
"datasetRegistries" : [ {
"id" : "osdu:dataset--File.Generic:null",
"kind" : "osdu:wks:dataset--File.Generic:1.0.0",
"acl" : {
"viewers" : [ "data.default.viewers@osdu.example.com" ],
"owners" : [ "data.default.owners@osdu.example.com" ]
},
"legal" : {
"otherRelevantDataCountries" : [ "US" ],
"status" : "compliant",
"legaltags" : [ "osdu-public-usa-dataset" ]
},
"data" : {
"DatasetProperties" : {
"FileSourceInfo" : {
"FileSource" : "s3://osr3m10-shared-561735291427-us-east-2-file-dms/osdu/X7g_6H3xwDwuYeYyfTdcmvD52Bo/",
"Name" : "energistics_well.xml"
}
},
"ResourceSecurityClassification" : "osdu:reference-data--ResourceSecurityClassification:RESTRICTED:",
"SchemaFormatTypeID" : "osdu:reference-data--SchemaFormatType:TabSeparatedColumnarText:"
}
} ]
}
Registration of dataset succeeded.
- Trigger DAG Workflow
POST https://r3m10.preshiptesting.osdu.aws/api/workflow/v1/workflow/Energistics_xml_ingest/workflowRun Given Authorization with Access/id token Given data-partition-id osdu Given Content-Type application/json
Given body :
{
"executionContext" : {
"Payload" : {
"AppKey" : "test-app",
"data-partition-id" : "osdu"
},
"Context" : {
"acl" : {
"viewers" : [ "data.default.viewers@osdu.example.com" ],
"owners" : [ "data.default.owners@osdu.example.com" ]
},
"legal" : {
"otherRelevantDataCountries" : [ "US" ],
"status" : "compliant",
"legaltags" : [ "osdu-public-usa-dataset" ]
},
"kind" : "osdu:wks:dataset--File.Generic:1.0.0",
"version" : 1.0,
"dataset_id" : "osdu:dataset--File.Generic:null",
"file_name" : "energistics_well.xml"
}
}
}
Response with run ID : 8e513e73-3ea8-46ce-b4c0-aa7a5bab441e
- Retrieving the status of the workflow 8e513e73-3ea8-46ce-b4c0-aa7a5bab441e
-
Given Authorization access/id token
-
Given data-partition-id : osdu
-
Status : submitted
- Retrieving the status of the workflow 8e513e73-3ea8-46ce-b4c0-aa7a5bab441e
-
Given Authorization access/id token
-
Given data-partition-id : osdu
-
Status : failed