Skip to content

GitLab

How to load a LAS data (Manifest based Ingestion) · Changes

Page history
Update How to load a LAS data (Manifest based Ingestion) authored by Chad Leong's avatar Chad Leong
Hide whitespace changes
Inline Side-by-side
How-to-load-a-LAS-data-(Manifest-based-Ingestion).md 0 → 100644
View page @ 8d0247f0
# Loading Data
## Manifest based loading with LAS file by Ivar Sørheim [Equinor]
### Generating Manifest
The manifest in this example was generated using the scripts provided here (bat files): https://community.opengroup.org/osdu/platform/open-test-data/-/tree/master/
### Prepping Manifest
Some changes might have to be done to the manifest in order to use it.
1. Replace legal tag
2. Replace 'otherRelevantCountries'
3. Replace Owners and Viewers
4. Fix invalid TimeDate
### Loading a well log
This is the process for loading a LAS well log into the OSDU Data platform.
Prerequisities:
* Reference data loaded, otherwise the ingest will fail on referential integrity
* Master data (wellbore) loaded, otherwise the ingest will fail on referential integrity
* Well Known Schemas from OSDU data definitions group loaded
* All core services up and running
```mermaid
graph TD;
Generate[Generate Well Log Manifest]-->WKS[Check if WKS exist];
Generate[Generate Well Log Manifest]--Manifest included in body to workflow request-->Workflow[Trigger workflow];
WKS[Check if WKS exist]--> Legal1[Get/Create Legal Tag];
Legal1[Get/Create Legal Tag]--Legal tag used in manifest-->Workflow[Trigger workflow];
Legal1[Get/Create Legal Tag]-->Upload[Get File upload signed URL];
Upload[Get File upload signed URL]--Filesource populated in Manifest-->Workflow;
Upload[Get File upload signed URL]-->Upload2[Upload LAS file];
Upload2-->Workflow[Trigger workflow];
Workflow-->Check[Check DAG status];
Check-->Search[Check ingested data by using search];
```
### Sending Manifest to workflow API
How to populate body when triggering workflow (insert: link to relevant postman collection):
{"executionContext": {\
"Payload": {\
"data-partition-id": "{{data-partition-id}}",\
"AppKey": "test-app"\
},\
"manifest": {your well log manifest goes here\
}\
} \
}
### Example populated manifest
```json
{
"kind": "{{data-partition-id}}:wks:Manifest:1.0.0",
"ReferenceData": [],
"MasterData": [],
"Data": {
"WorkProduct": {
"kind": "{{data-partition-id}}:wks:work-product--WorkProduct:1.0.0",
"acl": {
"owners": ["data.default.viewers@opendes.contoso.com"],
"viewers": ["data.default.owners@opendes.contoso.com"]
},
"legal": {
"legaltags": ["opendes-public-usa-dataset-7643990"],
"otherRelevantDataCountries": ["US"]
},
"data": {
"ResourceSecurityClassification": "{{data-partition-id}}:reference-data--ResourceSecurityClassification:RESTRICTED:",
"Name": "NPD-3145 LOG",
"Description": "Well Log",
"Components": [
"surrogate-key:wpc-1"
]
}
},
"WorkProductComponents": [
{
"id": "surrogate-key:wpc-1",
"kind": "{{data-partition-id}}:wks:work-product-component--WellLog:1.0.0",
"acl": {
"owners": ["data.default.viewers@opendes.contoso.com"],
"viewers": ["data.default.owners@opendes.contoso.com"]
},
"legal": {
"legaltags": ["opendes-public-usa-dataset-7643990"],
"otherRelevantDataCountries": ["US"]
},
"data": {
"ResourceSecurityClassification": "{{data-partition-id}}:reference-data--ResourceSecurityClassification:RESTRICTED:",
"Name": "NPD-3145 LOG",
"Description": "Well Log",
"Datasets": [
"surrogate-key:file-1"
],
"WellboreID": "{{data-partition-id}}:master-data--Wellbore:NPD-3145:",
"TopMeasuredDepth": 3666.5916,
"BottomMeasuredDepth": 4124.8584,
"Curves": [
{
"Mnemonic": "DEPTH",
"TopDepth": 3666.5916,
"BaseDepth": 4124.8584,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:"
},
{
"Mnemonic": "BS",
"TopDepth": 3666.5916,
"BaseDepth": 4124.8584,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:IN:"
},
{
"Mnemonic": "BVW",
"TopDepth": 3666.5916,
"BaseDepth": 3965.9052,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
},
{
"Mnemonic": "CARB_FLAG",
"TopDepth": 3666.5916,
"BaseDepth": 4124.8584,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:UNITLESS:"
},
{
"Mnemonic": "COAL_FLAG",
"TopDepth": 3666.5916,
"BaseDepth": 4124.8584,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:UNITLESS:"
},
{
"Mnemonic": "DELTA",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
},
{
"Mnemonic": "GRMAX",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:GAPI:"
},
{
"Mnemonic": "GRMIN",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:GAPI:"
},
{
"Mnemonic": "J",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:"
},
{
"Mnemonic": "KLHC_INT",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:MD:"
},
{
"Mnemonic": "KLOGH",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:MD:"
},
{
"Mnemonic": "M",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:UNITLESS:"
},
{
"Mnemonic": "N",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:"
},
{
"Mnemonic": "PERF_FLAG",
"TopDepth": 3824.9352,
"BaseDepth": 3958.1328,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:UNITLESS:"
},
{
"Mnemonic": "PHIF",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
},
{
"Mnemonic": "PORD",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
},
{
"Mnemonic": "RHOFL",
"TopDepth": 3666.5916,
"BaseDepth": 4124.8584,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:G%2FCM3:"
},
{
"Mnemonic": "RHOMA",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:G%2FCM3:"
},
{
"Mnemonic": "RW",
"TopDepth": 3666.5916,
"BaseDepth": 3965.9052,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:OHMM:"
},
{
"Mnemonic": "SAND_FLAG",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:UNITLESS:"
},
{
"Mnemonic": "SW",
"TopDepth": 3666.5916,
"BaseDepth": 3965.9052,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
},
{
"Mnemonic": "SWIRR",
"TopDepth": 3666.5916,
"BaseDepth": 3966.21,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:"
},
{
"Mnemonic": "TEMP",
"TopDepth": 3666.5916,
"BaseDepth": 3965.9052,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:DEGC:"
},
{
"Mnemonic": "VSH",
"TopDepth": 3666.5916,
"BaseDepth": 4124.8584,
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
}
]
}
}
],
"Datasets": [
{
"id": "surrogate-key:file-1",
"kind": "{{data-partition-id}}:wks:dataset--File.Generic:1.0.0",
"acl": {
"owners": ["data.default.viewers@opendes.contoso.com"],
"viewers": ["data.default.owners@opendes.contoso.com"]
},
"legal": {
"legaltags": ["opendes-public-usa-dataset-7643990"],
"otherRelevantDataCountries": ["US"]
},
"data": {
"ResourceSecurityClassification": "{{data-partition-id}}:reference-data--ResourceSecurityClassification:RESTRICTED:",
"SchemaFormatTypeID": "{{data-partition-id}}:reference-data--SchemaFormatType:LAS2:",
"DatasetProperties": {
"FileSourceInfo": {
"FileSource": "{{FileSource}}",
"PreloadFilePath": ""
}
}
}
}
]
}
}
}
}
```
### Searching ingested data
POST https://{{SEARCH_HOST}}/query
BODY
```
{
"kind": "{{data-partition-id}}:wks:work-product-component--WellLog:1.0.0",
"query": "*",
"limit": 100
}
```