Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
  • P Pre Shipping
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 33
    • Issues 33
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Open Subsurface Data Universe SoftwareOpen Subsurface Data Universe Software
  • Platform
  • Pre Shipping
  • Issues
  • #60
Closed
Open
Issue created Sep 22, 2021 by etienne peysson@epeyssonMaintainer

Witsml Parser (Trajectory) - DAG failure - R3M8

Following are the steps I've followed to get the error from Airflow (you'll see the error at the bottom)

  1. Get the SignedUrl from the File DMS

GET https://osdu-cpd-osdu.odi-osdu-og-fa7661852f2ab29a6be32f560b2f5573-0000.us-south.containers.appdomain.cloud/osdu-file/api/file/v2/files/uploadURL

  • Given Authorization access/id token

  • Given data-partition-id : opendes

  • Signed Url : https://minio-osdu-minio.odi-osdu-og-fa7661852f2ab29a6be32f560b2f5573-0000.us-south.containers.appdomain.cloud/oc-cpd-opendes-staging-bucket/dbbe4c019ef5476597b417574ffcdb6a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210922T121558Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86399&X-Amz-Credential=minio%2F20210922%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=97344e6d73c77e073e27dfe5942eb9744cfac56c4d96a6f6062b84e83691182e

  • File source ID : dbbe4c019ef5476597b417574ffcdb6a

  1. File upload

PUT https://minio-osdu-minio.odi-osdu-og-fa7661852f2ab29a6be32f560b2f5573-0000.us-south.containers.appdomain.cloud/oc-cpd-opendes-staging-bucket/dbbe4c019ef5476597b417574ffcdb6a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210922T121558Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86399&X-Amz-Credential=minio%2F20210922%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=97344e6d73c77e073e27dfe5942eb9744cfac56c4d96a6f6062b84e83691182e

  • Given headers : Content-Type: Application/xml
  • Given Binary file stream

Upload file response code 200

  1. Load metadata

POST https://osdu-cpd-osdu.odi-osdu-og-fa7661852f2ab29a6be32f560b2f5573-0000.us-south.containers.appdomain.cloud/osdu-file/api/file/v2/files/metadata

  • Given Authorization with Access/id token
  • Given data-partition-id opendes
  • Given Content-Type application/json
  • Given x-ms-blob-type BlockBlob

Given body :

{
  "data" : {
    "TotalSize" : 5299.0,
    "Source" : "TNO Data Source",
    "Name" : "Trajectory DC",
    "Endian" : "BIG",
    "Description" : "Trajectory WITSML dataset",
    "DatasetProperties" : {
      "FileSourceInfo" : {
        "FileSource" : "dbbe4c019ef5476597b417574ffcdb6a",
        "Name" : "trajectory_DC.xml",
        "PreloadFilePath" : "s3://oc-cpd-opendes-staging-bucket/dbbe4c019ef5476597b417574ffcdb6a",
        "PreloadFileCreateUser" : "preshipteama",
        "PreloadFileModifyDate" : "2021-09-22 02:15:59",
        "PreloadFileModifyUser" : "preshipteama"
      }
    }
  },
  "kind" : "opendes:wks:dataset--File.Generic:1.0.0",
  "acl" : {
    "viewers" : [ "data.default.viewers@opendes.ibm.com" ],
    "owners" : [ "data.default.owners@opendes.ibm.com" ]
  },
  "legal" : {
    "otherRelevantDataCountries" : [ "US" ],
    "status" : "compliant",
    "legaltags" : [ "opendes-Test-Legal-Tag-7292798" ]
  },
  "createUser" : "preshipteama",
  "createTime" : "2021-09-22 02:15:59",
  "modifyUser" : "preshipteama",
  "modifyTime" : "2021-09-22 02:15:59"
}

Response obtained with dataset ID opendes:dataset--File.Generic:3f2d8cb1-dfec-48e6-ba90-986b5a89a0ed

  1. Trigger DAG Workflow

POST https://osdu-cpd-osdu.odi-osdu-og-fa7661852f2ab29a6be32f560b2f5573-0000.us-south.containers.appdomain.cloud/osdu-workflow/api/workflow/v1/workflow/Energistics_xml_ingest/workflowRun Given Authorization with Access/id token Given data-partition-id opendes Given Content-Type application/json

Given body :

{
  "executionContext" : {
    "Payload" : {
      "AppKey" : "test-app",
      "data-partition-id" : "opendes"
    },
    "Context" : {
      "acl" : {
        "viewers" : [ "data.default.viewers@opendes.ibm.com" ],
        "owners" : [ "data.default.owners@opendes.ibm.com" ]
      },
      "legal" : {
        "otherRelevantDataCountries" : [ "US" ],
        "status" : "compliant",
        "legaltags" : [ "opendes-Test-Legal-Tag-7292798" ]
      },
      "kind" : "opendes:wks:dataset--File.Generic:1.0.0",
      "version" : 1.0,
      "dataset_id" : "opendes:dataset--File.Generic:3f2d8cb1-dfec-48e6-ba90-986b5a89a0ed",
      "file_name" : "trajectory_DC.xml",
      "preload_file_path" : "s3://oc-cpd-opendes-staging-bucket/dbbe4c019ef5476597b417574ffcdb6a"
    }
  }
}

Response with run ID : 0174ebc2-1c65-47b4-9c06-7ac5a8823925

Error logs :

[2021-09-22 12:16:49,662] {logging_mixin.py:112} INFO - [2021-09-22 12:16:49,662] {pod_launcher.py:142} INFO - Event: witsml-parser-task-25a574f5 had an event of type Running
[2021-09-22 12:16:59,071] {logging_mixin.py:112} INFO - [2021-09-22 12:16:59,071] {pod_launcher.py:125} INFO - b'IBMBlobStorageFactory().get_s3_client\n'
[2021-09-22 12:16:59,071] {logging_mixin.py:112} INFO - [2021-09-22 12:16:59,071] {pod_launcher.py:125} INFO - b'IBMBlobStorageFactory().s3_client:  <botocore.client.S3 object at 0x7fd4d46e21d0>\n'
[2021-09-22 12:16:59,072] {logging_mixin.py:112} INFO - [2021-09-22 12:16:59,072] {pod_launcher.py:125} INFO - b'Traceback (most recent call last):\n'
[2021-09-22 12:16:59,075] {logging_mixin.py:112} INFO - [2021-09-22 12:16:59,072] {pod_launcher.py:125} INFO - b'  File "main.py", line 45, in <module>\n'
[2021-09-22 12:16:59,075] {logging_mixin.py:112} INFO - [2021-09-22 12:16:59,075] {pod_launcher.py:125} INFO - b'    manifest = json.dumps(main(json.loads(args.context), args.file_service))\n'
[2021-09-22 12:16:59,076] {logging_mixin.py:112} INFO - [2021-09-22 12:16:59,076] {pod_launcher.py:125} INFO - b'  File "main.py", line 35, in main\n'
[2021-09-22 12:16:59,076] {logging_mixin.py:112} INFO - [2021-09-22 12:16:59,076] {pod_launcher.py:125} INFO - b'    manifest = manifest_creator.create_manifest(execution_context)\n'
[2021-09-22 12:16:59,076] {logging_mixin.py:112} INFO - [2021-09-22 12:16:59,076] {pod_launcher.py:125} INFO - b'  File "/home/witsml_parser/create_energistics_manifest.py", line 234, in create_manifest\n'
[2021-09-22 12:16:59,077] {logging_mixin.py:112} INFO - [2021-09-22 12:16:59,076] {pod_launcher.py:125} INFO - b'    dataset_dms_client = DatasetDmsClient(DefaultConfigManager(), self.payload_context.data_partition_id)\n'
[2021-09-22 12:16:59,077] {logging_mixin.py:112} INFO - [2021-09-22 12:16:59,077] {pod_launcher.py:125} INFO - b'TypeError: __init__() takes from 1 to 2 positional arguments but 3 were given\n'
[2021-09-22 12:17:02,050] {logging_mixin.py:112} INFO - [2021-09-22 12:17:02,049] {pod_launcher.py:217} INFO - Running command... cat /airflow/xcom/return.json

[2021-09-22 12:17:02,383] {logging_mixin.py:112} INFO - [2021-09-22 12:17:02,382] {pod_launcher.py:224} INFO - cat: can't open '/airflow/xcom/return.json': No such file or directory

More details here : http://airflow-web-odi-airflow-ns.odi-osdu-og-fa7661852f2ab29a6be32f560b2f5573-0000.us-south.containers.appdomain.cloud/log?task_id=witsml_parser_task&dag_id=Energistics_xml_ingest&execution_date=2021-09-22T12%3A16%3A06%2B00%3A00

Assignee
Assign to
Time tracking