Skip to content

GitLab

Last edited by Chad Leong
Page history

How to load a LAS data (Manifest based Ingestion)

Manifest based loading with LAS file by Ivar Sørheim [Equinor]

Generating Manifest

The manifest in this example was generated using the scripts provided here (bat files): https://community.opengroup.org/osdu/platform/open-test-data/-/tree/master/

Prepping Manifest

Some changes might have to be done to the manifest in order to use it.

  1. Replace legal tag
  2. Replace 'otherRelevantCountries'
  3. Replace Owners and Viewers
  4. Fix invalid TimeDate

Loading a well log

This is the process for loading a LAS well log into the OSDU Data platform.

Prerequisities:

  • Reference data loaded, otherwise the ingest will fail on referential integrity
  • Master data (wellbore) loaded, otherwise the ingest will fail on referential integrity
  • Well Known Schemas from OSDU data definitions group loaded
  • All core services up and running
graph TD;
  Generate[Generate Well Log Manifest]-->WKS[Check if WKS exist];
  Generate[Generate Well Log Manifest]--Manifest included in body to workflow request-->Workflow[Trigger workflow];
  WKS[Check if WKS exist]--> Legal1[Get/Create Legal Tag];
  Legal1[Get/Create Legal Tag]--Legal tag used in manifest-->Workflow[Trigger workflow];
  Legal1[Get/Create Legal Tag]-->Upload[Get File upload signed URL];
  Upload[Get File upload signed URL]--Filesource populated in Manifest-->Workflow;
  Upload[Get File upload signed URL]-->Upload2[Upload LAS file];
  Upload2-->Workflow[Trigger workflow];
  Workflow-->Check[Check DAG status];
  Check-->Search[Check ingested data by using search];

Sending Manifest to workflow API

How to populate body when triggering workflow (insert: link to relevant postman collection):

{"executionContext": {
"Payload": {
"data-partition-id": "{{data-partition-id}}",
"AppKey": "test-app"
},
"manifest": {your well log manifest goes here
}
}
}

Example populated manifest


{
    "kind": "{{data-partition-id}}:wks:Manifest:1.0.0",
    "ReferenceData": [],
    "MasterData": [],
    "Data": {
        "WorkProduct": {
            "kind": "{{data-partition-id}}:wks:work-product--WorkProduct:1.0.0",
            "acl": {
                    "owners": ["data.default.viewers@opendes.contoso.com"],
                    "viewers": ["data.default.owners@opendes.contoso.com"]
                },
            "legal": {
                "legaltags": ["opendes-public-usa-dataset-7643990"],
                "otherRelevantDataCountries": ["US"]
            },
            "data": {
                "ResourceSecurityClassification": "{{data-partition-id}}:reference-data--ResourceSecurityClassification:RESTRICTED:",
                "Name": "NPD-3145 LOG",
                "Description": "Well Log",
                "Components": [
                    "surrogate-key:wpc-1"
                ]
            }
        },
        "WorkProductComponents": [
            {
                "id": "surrogate-key:wpc-1",
                "kind": "{{data-partition-id}}:wks:work-product-component--WellLog:1.0.0",
                "acl": {
                    "owners": ["data.default.viewers@opendes.contoso.com"],
                    "viewers": ["data.default.owners@opendes.contoso.com"]
                },
                "legal": {
                    "legaltags": ["opendes-public-usa-dataset-7643990"],
                    "otherRelevantDataCountries": ["US"]
                },
                "data": {
                    "ResourceSecurityClassification": "{{data-partition-id}}:reference-data--ResourceSecurityClassification:RESTRICTED:",
                    "Name": "NPD-3145 LOG",
                    "Description": "Well Log",
                    "Datasets": [
                        "surrogate-key:file-1"
                    ],
                    "WellboreID": "{{data-partition-id}}:master-data--Wellbore:NPD-3145:",
                    "TopMeasuredDepth": 3666.5916,
                    "BottomMeasuredDepth": 4124.8584,
                    "Curves": [
                        {
                            "Mnemonic": "DEPTH",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 4124.8584,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
                            "CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:"
                        },
                        {
                            "Mnemonic": "BS",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 4124.8584,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
                            "CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:IN:"
                        },
                        {
                            "Mnemonic": "BVW",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 3965.9052,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
                            "CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
                        },
                        {
                            "Mnemonic": "CARB_FLAG",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 4124.8584,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
                            "CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:UNITLESS:"
                        },
                        {
                            "Mnemonic": "COAL_FLAG",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 4124.8584,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
                            "CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:UNITLESS:"
                        },
                        {
                            "Mnemonic": "DELTA",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 3966.21,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
                            "CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
                        },
                        {
                            "Mnemonic": "GRMAX",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 3966.21,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
                            "CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:GAPI:"
                        },
                        {
                            "Mnemonic": "GRMIN",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 3966.21,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
                            "CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:GAPI:"
                        },
                        {
                            "Mnemonic": "J",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 3966.21,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:"
                        },
                        {
                            "Mnemonic": "KLHC_INT",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 3966.21,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
                            "CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:MD:"
                        },
                        {
                            "Mnemonic": "KLOGH",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 3966.21,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
                            "CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:MD:"
                        },
                        {
                            "Mnemonic": "M",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 3966.21,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
                            "CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:UNITLESS:"
                        },
                        {
                            "Mnemonic": "N",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 3966.21,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:"
                        },
                        {
                            "Mnemonic": "PERF_FLAG",
                            "TopDepth": 3824.9352,
                            "BaseDepth": 3958.1328,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
                            "CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:UNITLESS:"
                        },
                        {
                            "Mnemonic": "PHIF",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 3966.21,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
                            "CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
                        },
                        {
                            "Mnemonic": "PORD",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 3966.21,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
                            "CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
                        },
                        {
                            "Mnemonic": "RHOFL",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 4124.8584,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
                            "CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:G%2FCM3:"
                        },
                        {
                            "Mnemonic": "RHOMA",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 3966.21,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
                            "CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:G%2FCM3:"
                        },
                        {
                            "Mnemonic": "RW",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 3965.9052,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
                            "CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:OHMM:"
                        },
                        {
                            "Mnemonic": "SAND_FLAG",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 3966.21,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
                            "CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:UNITLESS:"
                        },
                        {
                            "Mnemonic": "SW",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 3965.9052,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
                            "CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
                        },
                        {
                            "Mnemonic": "SWIRR",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 3966.21,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:"
                        },
                        {
                            "Mnemonic": "TEMP",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 3965.9052,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
                            "CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:DEGC:"
                        },
                        {
                            "Mnemonic": "VSH",
                            "TopDepth": 3666.5916,
                            "BaseDepth": 4124.8584,
                            "DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
                            "CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
                        }
                    ]
                }
            }
        ],
        "Datasets": [
            {
                "id": "surrogate-key:file-1",
                "kind": "{{data-partition-id}}:wks:dataset--File.Generic:1.0.0",
                "acl": {
                    "owners": ["data.default.viewers@opendes.contoso.com"],
                    "viewers": ["data.default.owners@opendes.contoso.com"]
                },
                "legal": {
                    "legaltags": ["opendes-public-usa-dataset-7643990"],
                    "otherRelevantDataCountries": ["US"]
                },
                "data": {
                    "ResourceSecurityClassification": "{{data-partition-id}}:reference-data--ResourceSecurityClassification:RESTRICTED:",
                    "SchemaFormatTypeID": "{{data-partition-id}}:reference-data--SchemaFormatType:LAS2:",
                    "DatasetProperties": {
                        "FileSourceInfo": {
                            "FileSource": "{{FileSource}}",
                            "PreloadFilePath": ""
                        }
                    }
                }
            }
        ]
    }
}
 }
 }

Searching ingested data

POST https://{{SEARCH_HOST}}/query

BODY

{
   "kind": "{{data-partition-id}}:wks:work-product-component--WellLog:1.0.0",
   "query": "*",
   "limit": 100
}