|
|
# Loading Data
|
|
|
|
|
|
## Manifest based loading with LAS file by Ivar Sørheim [Equinor]
|
|
|
|
|
|
### Generating Manifest
|
|
|
The manifest in this example was generated using the scripts provided here (bat files): https://community.opengroup.org/osdu/platform/open-test-data/-/tree/master/
|
|
|
|
|
|
### Prepping Manifest
|
|
|
Some changes might have to be done to the manifest in order to use it.
|
|
|
|
|
|
1. Replace legal tag
|
|
|
2. Replace 'otherRelevantCountries'
|
|
|
3. Replace Owners and Viewers
|
|
|
4. Fix invalid TimeDate
|
|
|
|
|
|
### Loading a well log
|
|
|
This is the process for loading a LAS well log into the OSDU Data platform.
|
|
|
|
|
|
Prerequisities:
|
|
|
* Reference data loaded, otherwise the ingest will fail on referential integrity
|
|
|
* Master data (wellbore) loaded, otherwise the ingest will fail on referential integrity
|
|
|
* Well Known Schemas from OSDU data definitions group loaded
|
|
|
* All core services up and running
|
|
|
|
|
|
```mermaid
|
|
|
graph TD;
|
|
|
Generate[Generate Well Log Manifest]-->WKS[Check if WKS exist];
|
|
|
Generate[Generate Well Log Manifest]--Manifest included in body to workflow request-->Workflow[Trigger workflow];
|
|
|
WKS[Check if WKS exist]--> Legal1[Get/Create Legal Tag];
|
|
|
Legal1[Get/Create Legal Tag]--Legal tag used in manifest-->Workflow[Trigger workflow];
|
|
|
Legal1[Get/Create Legal Tag]-->Upload[Get File upload signed URL];
|
|
|
Upload[Get File upload signed URL]--Filesource populated in Manifest-->Workflow;
|
|
|
Upload[Get File upload signed URL]-->Upload2[Upload LAS file];
|
|
|
Upload2-->Workflow[Trigger workflow];
|
|
|
Workflow-->Check[Check DAG status];
|
|
|
Check-->Search[Check ingested data by using search];
|
|
|
```
|
|
|
|
|
|
### Sending Manifest to workflow API
|
|
|
|
|
|
How to populate body when triggering workflow (insert: link to relevant postman collection):
|
|
|
|
|
|
{"executionContext": {\
|
|
|
"Payload": {\
|
|
|
"data-partition-id": "{{data-partition-id}}",\
|
|
|
"AppKey": "test-app"\
|
|
|
},\
|
|
|
"manifest": {your well log manifest goes here\
|
|
|
}\
|
|
|
} \
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Example populated manifest
|
|
|
|
|
|
```json
|
|
|
|
|
|
{
|
|
|
"kind": "{{data-partition-id}}:wks:Manifest:1.0.0",
|
|
|
"ReferenceData": [],
|
|
|
"MasterData": [],
|
|
|
"Data": {
|
|
|
"WorkProduct": {
|
|
|
"kind": "{{data-partition-id}}:wks:work-product--WorkProduct:1.0.0",
|
|
|
"acl": {
|
|
|
"owners": ["data.default.viewers@opendes.contoso.com"],
|
|
|
"viewers": ["data.default.owners@opendes.contoso.com"]
|
|
|
},
|
|
|
"legal": {
|
|
|
"legaltags": ["opendes-public-usa-dataset-7643990"],
|
|
|
"otherRelevantDataCountries": ["US"]
|
|
|
},
|
|
|
"data": {
|
|
|
"ResourceSecurityClassification": "{{data-partition-id}}:reference-data--ResourceSecurityClassification:RESTRICTED:",
|
|
|
"Name": "NPD-3145 LOG",
|
|
|
"Description": "Well Log",
|
|
|
"Components": [
|
|
|
"surrogate-key:wpc-1"
|
|
|
]
|
|
|
}
|
|
|
},
|
|
|
"WorkProductComponents": [
|
|
|
{
|
|
|
"id": "surrogate-key:wpc-1",
|
|
|
"kind": "{{data-partition-id}}:wks:work-product-component--WellLog:1.0.0",
|
|
|
"acl": {
|
|
|
"owners": ["data.default.viewers@opendes.contoso.com"],
|
|
|
"viewers": ["data.default.owners@opendes.contoso.com"]
|
|
|
},
|
|
|
"legal": {
|
|
|
"legaltags": ["opendes-public-usa-dataset-7643990"],
|
|
|
"otherRelevantDataCountries": ["US"]
|
|
|
},
|
|
|
"data": {
|
|
|
"ResourceSecurityClassification": "{{data-partition-id}}:reference-data--ResourceSecurityClassification:RESTRICTED:",
|
|
|
"Name": "NPD-3145 LOG",
|
|
|
"Description": "Well Log",
|
|
|
"Datasets": [
|
|
|
"surrogate-key:file-1"
|
|
|
],
|
|
|
"WellboreID": "{{data-partition-id}}:master-data--Wellbore:NPD-3145:",
|
|
|
"TopMeasuredDepth": 3666.5916,
|
|
|
"BottomMeasuredDepth": 4124.8584,
|
|
|
"Curves": [
|
|
|
{
|
|
|
"Mnemonic": "DEPTH",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 4124.8584,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
|
|
|
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "BS",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 4124.8584,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
|
|
|
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:IN:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "BVW",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 3965.9052,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
|
|
|
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "CARB_FLAG",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 4124.8584,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
|
|
|
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:UNITLESS:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "COAL_FLAG",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 4124.8584,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
|
|
|
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:UNITLESS:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "DELTA",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 3966.21,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
|
|
|
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "GRMAX",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 3966.21,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
|
|
|
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:GAPI:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "GRMIN",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 3966.21,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
|
|
|
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:GAPI:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "J",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 3966.21,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "KLHC_INT",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 3966.21,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
|
|
|
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:MD:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "KLOGH",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 3966.21,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
|
|
|
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:MD:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "M",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 3966.21,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
|
|
|
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:UNITLESS:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "N",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 3966.21,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "PERF_FLAG",
|
|
|
"TopDepth": 3824.9352,
|
|
|
"BaseDepth": 3958.1328,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
|
|
|
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:UNITLESS:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "PHIF",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 3966.21,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
|
|
|
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "PORD",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 3966.21,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
|
|
|
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "RHOFL",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 4124.8584,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
|
|
|
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:G%2FCM3:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "RHOMA",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 3966.21,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
|
|
|
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:G%2FCM3:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "RW",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 3965.9052,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
|
|
|
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:OHMM:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "SAND_FLAG",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 3966.21,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
|
|
|
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:UNITLESS:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "SW",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 3965.9052,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
|
|
|
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "SWIRR",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 3966.21,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "TEMP",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 3965.9052,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
|
|
|
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:DEGC:"
|
|
|
},
|
|
|
{
|
|
|
"Mnemonic": "VSH",
|
|
|
"TopDepth": 3666.5916,
|
|
|
"BaseDepth": 4124.8584,
|
|
|
"DepthUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:",
|
|
|
"CurveUnit": "{{data-partition-id}}:reference-data--UnitOfMeasure:V%2FV:"
|
|
|
}
|
|
|
]
|
|
|
}
|
|
|
}
|
|
|
],
|
|
|
"Datasets": [
|
|
|
{
|
|
|
"id": "surrogate-key:file-1",
|
|
|
"kind": "{{data-partition-id}}:wks:dataset--File.Generic:1.0.0",
|
|
|
"acl": {
|
|
|
"owners": ["data.default.viewers@opendes.contoso.com"],
|
|
|
"viewers": ["data.default.owners@opendes.contoso.com"]
|
|
|
},
|
|
|
"legal": {
|
|
|
"legaltags": ["opendes-public-usa-dataset-7643990"],
|
|
|
"otherRelevantDataCountries": ["US"]
|
|
|
},
|
|
|
"data": {
|
|
|
"ResourceSecurityClassification": "{{data-partition-id}}:reference-data--ResourceSecurityClassification:RESTRICTED:",
|
|
|
"SchemaFormatTypeID": "{{data-partition-id}}:reference-data--SchemaFormatType:LAS2:",
|
|
|
"DatasetProperties": {
|
|
|
"FileSourceInfo": {
|
|
|
"FileSource": "{{FileSource}}",
|
|
|
"PreloadFilePath": ""
|
|
|
}
|
|
|
}
|
|
|
}
|
|
|
}
|
|
|
]
|
|
|
}
|
|
|
}
|
|
|
}
|
|
|
}
|
|
|
|
|
|
```
|
|
|
|
|
|
### Searching ingested data
|
|
|
|
|
|
POST https://{{SEARCH_HOST}}/query
|
|
|
|
|
|
BODY
|
|
|
|
|
|
```
|
|
|
{
|
|
|
"kind": "{{data-partition-id}}:wks:work-product-component--WellLog:1.0.0",
|
|
|
"query": "*",
|
|
|
"limit": 100
|
|
|
}
|
|
|
``` |