Skip to content

GitLab

  • Menu
Projects Groups Snippets
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in / Register
  • D Data Loading
  • Subgroup information
    • Subgroup information
    • Activity
    • Labels
    • Members
  • Epics 3
    • Epics 3
    • List
    • Boards
    • Roadmap
  • Issues 29
    • Issues 29
    • List
    • Boards
    • Milestones
    • Iterations
  • Merge requests 1
    • Merge requests 1
  • Security & Compliance
    • Security & Compliance
    • Security dashboard
    • Vulnerability report
  • Packages & Registries
    • Packages & Registries
    • Package Registry
  • Analytics
    • Analytics
    • Insights
    • Issue
  • Wiki
    • Wiki
Collapse sidebar
  • Open Subsurface Data Universe Software
  • Platform
  • Data Flow
  • Data Loading
  • Wiki
  • How to load a LAS data (Manifest based Ingestion)

How to load a LAS data (Manifest based Ingestion) · Changes

Page history
Update How to load a LAS data (Manifest based Ingestion) authored Sep 07, 2021 by Chad Leong's avatar Chad Leong
Hide whitespace changes
Inline Side-by-side
How-to-load-a-LAS-data-(Manifest-based-Ingestion).md 0 → 100644
View page @ 8d0247f0
# Loading Data
## Manifest based loading with LAS file by Ivar Sørheim [Equinor]
### Generating Manifest
The manifest in this example was generated using the scripts provided here (bat files): https://community.opengroup.org/osdu/platform/open-test-data/-/tree/master/
### Prepping Manifest
Some changes might have to be done to the manifest in order to use it.
1. Replace legal tag
2. Replace 'otherRelevantCountries'
3. Replace Owners and Viewers
4. Fix invalid TimeDate
### Loading a well log
This is the process for loading a LAS well log into the OSDU Data platform.
Prerequisities:
* Reference data loaded, otherwise the ingest will fail on referential integrity
* Master data (wellbore) loaded, otherwise the ingest will fail on referential integrity
* Well Known Schemas from OSDU data definitions group loaded
* All core services up and running
```mermaid
graph TD;
Generate[Generate Well Log Manifest]-->WKS[Check if WKS exist];
Generate[Generate Well Log Manifest]--Manifest included in body to workflow request-->Workflow[Trigger workflow];
WKS[Check if WKS exist]--> Legal1[Get/Create Legal Tag];
Legal1[Get/Create Legal Tag]--Legal tag used in manifest-->Workflow[Trigger workflow];
Legal1[Get/Create Legal Tag]-->Upload[Get File upload signed URL];
Upload[Get File upload signed URL]--Filesource populated in Manifest-->Workflow;
Upload[Get File upload signed URL]-->Upload2[Upload LAS file];
Upload2-->Workflow[Trigger workflow];
Workflow-->Check[Check DAG status];
Check-->Search[Check ingested data by using search];
```
### Sending Manifest to workflow API
How to populate body when triggering workflow (insert: link to relevant postman collection):
{"executionContext": {\
"Payload": {\
"data-partition-id": "{{data-partition-id}}",\
"AppKey": "test-app"\
},\
"manifest": {your well log manifest goes here\
}\
} \
}
### Example populated manifest
```json
{
"kind": "{{data-partition-id}}:wks:Manifest:1.0.0",
"ReferenceData": [],
"MasterData": [],
"Data": {
"WorkProduct": {
"kind": "{{data-partition-id}}:wks:work-product--WorkProduct:1.0.0",
"acl": {
"owners": ["data.default.viewers@opendes.contoso.com"],
"viewers": ["data.default.owners@opendes.contoso.com"]
},
"legal": {
"legaltags": ["opendes-public-usa-dataset-7643990"],
"otherRelevantDataCountries": ["US"]
},
"data": {
"ResourceSecurityClassification": "{{data-partition-id}}:reference-data--ResourceSecurityClassification:RESTRICTED:",
"Name": "NPD-3145 LOG",
"Description": "Well Log",
"Components": [
"surrogate-key:wpc-1"
]
}
},
"WorkProductComponents": [
{
"id": "surrogate-key:wpc-1",
"kind": "{{data-partition-id}}:wks:work-product-component--WellLog:1.0.0",
"acl": {
"owners": ["data.default.viewers@opendes.contoso.com"],
"viewers": ["data.default.owners@opendes.contoso.com"]
},
"legal": {
"legaltags": ["opendes-public-usa-dataset-7643990"],
"otherRelevantDataCountries": ["US"]
},
"data": {
"ResourceSecurityClassification": "{{data-partition-id}}:reference-data--ResourceSecurityClassification:RESTRICTED:",
"Name": "NPD-3145 LOG",
"Description": "Well Log",
"Datasets": [
"surrogate-key:file-1"
],
"WellboreID": "{{data-partition-id}}:master-data--Wellbore:NPD-3145:",
"TopMeasuredDepth": 3666.5916,
"BottomMeasuredDepth": 4124.8584,
"Curves": [
{
"Mnemonic": "DEPTH",
"TopDepth": 3666.5916,
"BaseDepth": 4124.8584,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:"
},
{
"Mnemonic": "BS",
"TopDepth": 3666.5916,
"BaseDepth": 4124.8584,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:IN:"
},
{
"Mnemonic": "BVW",
"TopDepth": 3666.5916,
"BaseDepth": 3965.9052,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
},
{
"Mnemonic": "CARB_FLAG",
"TopDepth": 3666.5916,
"BaseDepth": 4124.8584,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:UNITLESS:"
},
{
"Mnemonic": "COAL_FLAG",
"TopDepth": 3666.5916,
"BaseDepth": 4124.8584,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:UNITLESS:"
},
{
"Mnemonic": "DELTA",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
},
{
"Mnemonic": "GRMAX",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:GAPI:"
},
{
"Mnemonic": "GRMIN",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:GAPI:"
},
{
"Mnemonic": "J",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:"
},
{
"Mnemonic": "KLHC_INT",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:MD:"
},
{
"Mnemonic": "KLOGH",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:MD:"
},
{
"Mnemonic": "M",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:UNITLESS:"
},
{
"Mnemonic": "N",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:"
},
{
"Mnemonic": "PERF_FLAG",
"TopDepth": 3824.9352,
"BaseDepth": 3958.1328,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:UNITLESS:"
},
{
"Mnemonic": "PHIF",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
},
{
"Mnemonic": "PORD",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
},
{
"Mnemonic": "RHOFL",
"TopDepth": 3666.5916,
"BaseDepth": 4124.8584,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:G%2FCM3:"
},
{
"Mnemonic": "RHOMA",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:G%2FCM3:"
},
{
"Mnemonic": "RW",
"TopDepth": 3666.5916,
"BaseDepth": 3965.9052,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:OHMM:"
},
{
"Mnemonic": "SAND_FLAG",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:UNITLESS:"
},
{
"Mnemonic": "SW",
"TopDepth": 3666.5916,
"BaseDepth": 3965.9052,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
},
{
"Mnemonic": "SWIRR",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:"
},
{
"Mnemonic": "TEMP",
"TopDepth": 3666.5916,
"BaseDepth": 3965.9052,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:DEGC:"
},
{
"Mnemonic": "VSH",
"TopDepth": 3666.5916,
"BaseDepth": 4124.8584,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
}
]
}
}
],
"Datasets": [
{
"id": "surrogate-key:file-1",
"kind": "{{data-partition-id}}:wks:dataset--File.Generic:1.0.0",
"acl": {
"owners": ["data.default.viewers@opendes.contoso.com"],
"viewers": ["data.default.owners@opendes.contoso.com"]
},
"legal": {
"legaltags": ["opendes-public-usa-dataset-7643990"],
"otherRelevantDataCountries": ["US"]
},
"data": {
"ResourceSecurityClassification": "{{data-partition-id}}:reference-data--ResourceSecurityClassification:RESTRICTED:",
"SchemaFormatTypeID": "{{data-partition-id}}:reference-data--SchemaFormatType:LAS2:",
"DatasetProperties": {
"FileSourceInfo": {
"FileSource": "{{FileSource}}",
"PreloadFilePath": ""
}
}
}
}
]
}
}
}
}
```
### Searching ingested data
POST https://{{SEARCH_HOST}}/query
BODY
```
{
"kind": "{{data-partition-id}}:wks:work-product-component--WellLog:1.0.0",
"query": "*",
"limit": 100
}
```
Clone repository
  • CSV Ingestion File Service
  • CSV Ingestion
  • Home
  • How to check for error in Airflow DAG
  • How to generate manifests using scripts
  • How to load a LAS data (Manifest based Ingestion)
  • How to load a WITSML data (WITSML Parser)
  • How to load a generic file in OSDU using File or Dataset Service
  • How to load a generic file in OSDU using File
    • How to load a generic file in OSDU using File Dataset service
  • Step by Step of Setting up Python Environment for sdutil in Windows
  • Troubleshooting Index Status of Data Ingested