Witsml Parser (Trajectory) - DAG failure - R3M8
Following are the steps I've followed to get the error from Airflow (you'll see the error at the bottom)
- Get the SignedUrl from the File DMS
-
Given Authorization access/id token
-
Given data-partition-id : opendes
-
File source ID : dbbe4c019ef5476597b417574ffcdb6a
- File upload
- Given headers : Content-Type: Application/xml
- Given Binary file stream
Upload file response code 200
- Load metadata
- Given Authorization with Access/id token
- Given data-partition-id opendes
- Given Content-Type application/json
- Given x-ms-blob-type BlockBlob
Given body :
{
"data" : {
"TotalSize" : 5299.0,
"Source" : "TNO Data Source",
"Name" : "Trajectory DC",
"Endian" : "BIG",
"Description" : "Trajectory WITSML dataset",
"DatasetProperties" : {
"FileSourceInfo" : {
"FileSource" : "dbbe4c019ef5476597b417574ffcdb6a",
"Name" : "trajectory_DC.xml",
"PreloadFilePath" : "s3://oc-cpd-opendes-staging-bucket/dbbe4c019ef5476597b417574ffcdb6a",
"PreloadFileCreateUser" : "preshipteama",
"PreloadFileModifyDate" : "2021-09-22 02:15:59",
"PreloadFileModifyUser" : "preshipteama"
}
}
},
"kind" : "opendes:wks:dataset--File.Generic:1.0.0",
"acl" : {
"viewers" : [ "data.default.viewers@opendes.ibm.com" ],
"owners" : [ "data.default.owners@opendes.ibm.com" ]
},
"legal" : {
"otherRelevantDataCountries" : [ "US" ],
"status" : "compliant",
"legaltags" : [ "opendes-Test-Legal-Tag-7292798" ]
},
"createUser" : "preshipteama",
"createTime" : "2021-09-22 02:15:59",
"modifyUser" : "preshipteama",
"modifyTime" : "2021-09-22 02:15:59"
}Response obtained with dataset ID opendes:dataset--File.Generic:3f2d8cb1-dfec-48e6-ba90-986b5a89a0ed
- Trigger DAG Workflow
POST https://osdu-cpd-osdu.odi-osdu-og-fa7661852f2ab29a6be32f560b2f5573-0000.us-south.containers.appdomain.cloud/osdu-workflow/api/workflow/v1/workflow/Energistics_xml_ingest/workflowRun Given Authorization with Access/id token Given data-partition-id opendes Given Content-Type application/json
Given body :
{
"executionContext" : {
"Payload" : {
"AppKey" : "test-app",
"data-partition-id" : "opendes"
},
"Context" : {
"acl" : {
"viewers" : [ "data.default.viewers@opendes.ibm.com" ],
"owners" : [ "data.default.owners@opendes.ibm.com" ]
},
"legal" : {
"otherRelevantDataCountries" : [ "US" ],
"status" : "compliant",
"legaltags" : [ "opendes-Test-Legal-Tag-7292798" ]
},
"kind" : "opendes:wks:dataset--File.Generic:1.0.0",
"version" : 1.0,
"dataset_id" : "opendes:dataset--File.Generic:3f2d8cb1-dfec-48e6-ba90-986b5a89a0ed",
"file_name" : "trajectory_DC.xml",
"preload_file_path" : "s3://oc-cpd-opendes-staging-bucket/dbbe4c019ef5476597b417574ffcdb6a"
}
}
}Response with run ID : 0174ebc2-1c65-47b4-9c06-7ac5a8823925
Error logs :
[2021-09-22 12:16:49,662] {logging_mixin.py:112} INFO - [2021-09-22 12:16:49,662] {pod_launcher.py:142} INFO - Event: witsml-parser-task-25a574f5 had an event of type Running
[2021-09-22 12:16:59,071] {logging_mixin.py:112} INFO - [2021-09-22 12:16:59,071] {pod_launcher.py:125} INFO - b'IBMBlobStorageFactory().get_s3_client\n'
[2021-09-22 12:16:59,071] {logging_mixin.py:112} INFO - [2021-09-22 12:16:59,071] {pod_launcher.py:125} INFO - b'IBMBlobStorageFactory().s3_client: <botocore.client.S3 object at 0x7fd4d46e21d0>\n'
[2021-09-22 12:16:59,072] {logging_mixin.py:112} INFO - [2021-09-22 12:16:59,072] {pod_launcher.py:125} INFO - b'Traceback (most recent call last):\n'
[2021-09-22 12:16:59,075] {logging_mixin.py:112} INFO - [2021-09-22 12:16:59,072] {pod_launcher.py:125} INFO - b' File "main.py", line 45, in <module>\n'
[2021-09-22 12:16:59,075] {logging_mixin.py:112} INFO - [2021-09-22 12:16:59,075] {pod_launcher.py:125} INFO - b' manifest = json.dumps(main(json.loads(args.context), args.file_service))\n'
[2021-09-22 12:16:59,076] {logging_mixin.py:112} INFO - [2021-09-22 12:16:59,076] {pod_launcher.py:125} INFO - b' File "main.py", line 35, in main\n'
[2021-09-22 12:16:59,076] {logging_mixin.py:112} INFO - [2021-09-22 12:16:59,076] {pod_launcher.py:125} INFO - b' manifest = manifest_creator.create_manifest(execution_context)\n'
[2021-09-22 12:16:59,076] {logging_mixin.py:112} INFO - [2021-09-22 12:16:59,076] {pod_launcher.py:125} INFO - b' File "/home/witsml_parser/create_energistics_manifest.py", line 234, in create_manifest\n'
[2021-09-22 12:16:59,077] {logging_mixin.py:112} INFO - [2021-09-22 12:16:59,076] {pod_launcher.py:125} INFO - b' dataset_dms_client = DatasetDmsClient(DefaultConfigManager(), self.payload_context.data_partition_id)\n'
[2021-09-22 12:16:59,077] {logging_mixin.py:112} INFO - [2021-09-22 12:16:59,077] {pod_launcher.py:125} INFO - b'TypeError: __init__() takes from 1 to 2 positional arguments but 3 were given\n'
[2021-09-22 12:17:02,050] {logging_mixin.py:112} INFO - [2021-09-22 12:17:02,049] {pod_launcher.py:217} INFO - Running command... cat /airflow/xcom/return.json
[2021-09-22 12:17:02,383] {logging_mixin.py:112} INFO - [2021-09-22 12:17:02,382] {pod_launcher.py:224} INFO - cat: can't open '/airflow/xcom/return.json': No such file or directory