AWS - key error partition_url - Witsml parser

Here is the whole process but it's likely an Airflow configuration issue :

Full report :

About to upload the following file : energistics_well.xml

Access/ID Token retrieval

Semi Manual flow from AWS

File upload

Get the SignedUrl from the Dataset API

GET https://r3.../getStorageInstructions?kindSubType=dataset--File.Generic

  • Given Authorization access/id token

  • Given data-partition-id : osdu

  • Unsigned Url : s3://osr3m10-sh... Need to submit metadata ? or register ! for dataset (using unsigned url)

File upload

PUT https://osr3m10...

  • Given headers :

Content-Type: Application/xml

  • Given Binary file stream

Upload file response code 200

File registration

Given body :

{
  "datasetRegistries" : [ {
    "id" : "osdu:dataset--File.Generic:null",
    "kind" : "osdu:wks:dataset--File.Generic:1.0.0",
    "acl" : {
      "viewers" : [ "data.default.viewers@osdu.example.com" ],
      "owners" : [ "data.default.owners@osdu.example.com" ]
    },
    "legal" : {
      "otherRelevantDataCountries" : [ "US" ],
      "status" : "compliant",
      "legaltags" : [ "osdu-public-usa-dataset" ]
    },
    "data" : {
      "DatasetProperties" : {
        "FileSourceInfo" : {
          "FileSource" : "s3://osr3m10-shared-561735291427-us-east-2-file-dms/osdu/X7g_6H3xwDwuYeYyfTdcmvD52Bo/",
          "Name" : "energistics_well.xml"
        }
      },
      "ResourceSecurityClassification" : "osdu:reference-data--ResourceSecurityClassification:RESTRICTED:",
      "SchemaFormatTypeID" : "osdu:reference-data--SchemaFormatType:TabSeparatedColumnarText:"
    }
  } ]
}

Registration of dataset succeeded.

  1. Trigger DAG Workflow

POST https://r3m10.preshiptesting.osdu.aws/api/workflow/v1/workflow/Energistics_xml_ingest/workflowRun Given Authorization with Access/id token Given data-partition-id osdu Given Content-Type application/json

Given body :

{
  "executionContext" : {
    "Payload" : {
      "AppKey" : "test-app",
      "data-partition-id" : "osdu"
    },
    "Context" : {
      "acl" : {
        "viewers" : [ "data.default.viewers@osdu.example.com" ],
        "owners" : [ "data.default.owners@osdu.example.com" ]
      },
      "legal" : {
        "otherRelevantDataCountries" : [ "US" ],
        "status" : "compliant",
        "legaltags" : [ "osdu-public-usa-dataset" ]
      },
      "kind" : "osdu:wks:dataset--File.Generic:1.0.0",
      "version" : 1.0,
      "dataset_id" : "osdu:dataset--File.Generic:null",
      "file_name" : "energistics_well.xml"
    }
  }
}

Response with run ID : 8e513e73-3ea8-46ce-b4c0-aa7a5bab441e

  1. Retrieving the status of the workflow 8e513e73-3ea8-46ce-b4c0-aa7a5bab441e

GET https://r3m10.preshiptesting.osdu.aws/api/workflow/v1/workflow/Energistics_xml_ingest/workflowRun/8e513e73-3ea8-46ce-b4c0-aa7a5bab441e

  • Given Authorization access/id token

  • Given data-partition-id : osdu

  • Status : submitted

  1. Retrieving the status of the workflow 8e513e73-3ea8-46ce-b4c0-aa7a5bab441e

GET https://r3m10.preshiptesting.osdu.aws/api/workflow/v1/workflow/Energistics_xml_ingest/workflowRun/8e513e73-3ea8-46ce-b4c0-aa7a5bab441e

  • Given Authorization access/id token

  • Given data-partition-id : osdu

  • Status : failed

Error : image