Changes

Page history
Update OSDU® Data Platform Data Loading Quick Start Guide authored by Chad Leong's avatar Chad Leong
Show whitespace changes
Inline Side-by-side
OSDU-Data-Platform-Data-Loading-Quick-Start-Guide.md
View page @ bb24d3fb
--- ---
title: OSDU® Data Platform Data Loading Quick Start Guide title: OSDU® Data Platform Data Loading Quick Start Guide
--- ---
# OSDU® Data Platform - Data Loading Quick Start Guide # OSDU® Data Platform - Data Loading Quick Start Guide
## Contents ## Contents
...@@ -21,8 +20,9 @@ These are the important links/documents that, you should first try to read and b ...@@ -21,8 +20,9 @@ These are the important links/documents that, you should first try to read and b
3. [OSDU Core Services](https://community.opengroup.org/groups/osdu/platform/-/wikis/Core-Services-Overview) - This contains the latest core services API and documentation that the OSDU® platform supports. 3. [OSDU Core Services](https://community.opengroup.org/groups/osdu/platform/-/wikis/Core-Services-Overview) - This contains the latest core services API and documentation that the OSDU® platform supports.
## Terms and Acronyms ## Terms and Acronyms
| Term | Description | | Term | Description |
|------|-------------| | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Airflow | Airflow is the designated workflow engine for OSDU®. Airflow is used to schedule and orchestrate the different workflows in OSDU® for data flow. Best practices can be found [here](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/wikis/Ingestion-DAG-Best-Practices). | | Airflow | Airflow is the designated workflow engine for OSDU®. Airflow is used to schedule and orchestrate the different workflows in OSDU® for data flow. Best practices can be found [here](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/wikis/Ingestion-DAG-Best-Practices). |
| Manifest | A manifest is a container specifically designed for facilitating metadata into the OSDU® platform. As of this writing, a Manifest has structures to support holding metadata records of the following types: Reference Data Master Data Work Product Work Product Component Datasets | | Manifest | A manifest is a container specifically designed for facilitating metadata into the OSDU® platform. As of this writing, a Manifest has structures to support holding metadata records of the following types: Reference Data Master Data Work Product Work Product Component Datasets |
| Source Data | Source Data might be an Excel file, LAS/DLIS files, Seismic data, text files, data streams, databases, etc. One of the goals of OSDU® is to support storing source data in its original format to preserve lineage. Metadata is created to allow this source data to remain in its original format yet remain searchable and discoverable within the platform. | | Source Data | Source Data might be an Excel file, LAS/DLIS files, Seismic data, text files, data streams, databases, etc. One of the goals of OSDU® is to support storing source data in its original format to preserve lineage. Metadata is created to allow this source data to remain in its original format yet remain searchable and discoverable within the platform. |
...@@ -40,12 +40,12 @@ This document addresses end-to-end data loading from the perspective of the end- ...@@ -40,12 +40,12 @@ This document addresses end-to-end data loading from the perspective of the end-
The Data Flow workstream covers the full end-to-end process of facilitating data into the OSDU® platform. The Data Ingestion workstream is part of the Data Loading workstream. The Data Flow workstream covers the full end-to-end process of facilitating data into the OSDU® platform. The Data Ingestion workstream is part of the Data Loading workstream.
* Loading – this workstream captures all the work necessary to ready the data for ingestion. Activities might include: - Loading – this workstream captures all the work necessary to ready the data for ingestion. Activities might include:
* Data fetching and organization - Data fetching and organization
* Data massaging and formatting - Data massaging and formatting
* Manifest/Metadata creation - Manifest/Metadata creation
* Loading source data to a landing zone or staging area - Loading source data to a landing zone or staging area
* Ingestion – this workstream facilitates data into the OSDU® Data Platform. There are several data ingestion services available. - Ingestion – this workstream facilitates data into the OSDU® Data Platform. There are several data ingestion services available.
### Data Platform Overview (Simplified) ### Data Platform Overview (Simplified)
...@@ -60,22 +60,22 @@ Below is a simplified overview of the data platform. There are more interactions ...@@ -60,22 +60,22 @@ Below is a simplified overview of the data platform. There are more interactions
These are the main OSDU® data types: These are the main OSDU® data types:
1. [Reference Data](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/ReferenceValues/Manifests/reference-data) - These are the standard naming for the data values. For example, the reference value for measured depth is MD and for elevation is ELEV. Whenever these values are being used, the reference data must be first loaded in the OSDU® platform. There are 3 governance levels for the reference data: 1. [Reference Data](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/ReferenceValues/Manifests/reference-data) - These are the standard naming for the data values. For example, the reference value for measured depth is MD and for elevation is ELEV. Whenever these values are being used, the reference data must be first loaded in the OSDU® platform. There are 3 governance levels for the reference data:
* [Fixed](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/ReferenceValues/Manifests/reference-data/FIXED) - Pre-determined by agreement in OSDU® Forum and shall not be changed. This allows interoperability between companies. - [Fixed](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/ReferenceValues/Manifests/reference-data/FIXED) - Pre-determined by agreement in OSDU® Forum and shall not be changed. This allows interoperability between companies.
* [Open](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/ReferenceValues/Manifests/reference-data/OPEN) - Agreed by OSDU® Forum but companies may extend with custom values. Custom values shall not conflict with Forum values. This allows some level of interoperability between companies. - [Open](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/ReferenceValues/Manifests/reference-data/OPEN) - Agreed by OSDU® Forum but companies may extend with custom values. Custom values shall not conflict with Forum values. This allows some level of interoperability between companies.
* [Local](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/ReferenceValues/Manifests/reference-data/LOCAL) - OSDU® Forum makes no declaration about the values and companies need to create their own list. This list does not benefit much from interoperability and agreed-upon values are hard to come by. - [Local](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/ReferenceValues/Manifests/reference-data/LOCAL) - OSDU® Forum makes no declaration about the values and companies need to create their own list. This list does not benefit much from interoperability and agreed-upon values are hard to come by.
2. [Master Data](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Generated/master-data) - A record of the information about business objects that we manage in the OSDU® record catalog. For example, a list of field names with well names and their associated wellbore names. 2. [Master Data](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Generated/master-data) - A record of the information about business objects that we manage in the OSDU® record catalog. For example, a list of field names with well names and their associated wellbore names.
3. [Work Product](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Generated/work-product) - A record that ties together a set of work product components such as a group of well logs inside a wellbore. 3. [Work Product](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Generated/work-product) - A record that ties together a set of work product components such as a group of well logs inside a wellbore.
4. [Work Product Components](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Generated/work-product-component) - A record that describes the business content of a single well log, such as the log data information, top, bottom depth of the well log. 4. [Work Product Components](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Generated/work-product-component) - A record that describes the business content of a single well log, such as the log data information, top, bottom depth of the well log.
* Here is the [list](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/E-R/work-product-component#supported-bulk-standards) of the supported bulk standards in OSDU®. - Here is the [list](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/E-R/work-product-component#supported-bulk-standards) of the supported bulk standards in OSDU®.
5. [File](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Generated/dataset) - A record that describes the metadata about the digital files, but does not describe the business content of the file, such as the file size, checksum of a well log. 5. [File](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Generated/dataset) - A record that describes the metadata about the digital files, but does not describe the business content of the file, such as the file size, checksum of a well log.
### Getting the data into the OSDU® Data Platform ### Getting the data into the OSDU® Data Platform
Once you are familiar with the OSDU® data types, you must understand that there are several [data flow services](https://community.opengroup.org/groups/osdu/platform/-/wikis/Core-Services-Overview#data-flow-services-and-dags) to bring the data into the OSDU® Data Platform. There are pros & cons in each approach as detailed in each section link below. Once you are familiar with the OSDU® data types, you must understand that there are several [data flow services](https://community.opengroup.org/groups/osdu/platform/-/wikis/Core-Services-Overview#data-flow-services-and-dags) to bring the data into the OSDU® Data Platform. There are pros & cons in each approach as detailed in each section link below.
* [CSV Parser Ingestion](https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser) – Schlumberger has developed an ingestion workflow capable of parsing a CSV file into a schema and loading each entry as a record into OSDU®. There is future work to enrich the flattened schema structure created by the CSV parser into an R3-style schema. - [CSV Parser Ingestion](https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser) – Schlumberger has developed an ingestion workflow capable of parsing a CSV file into a schema and loading each entry as a record into OSDU®. There is future work to enrich the flattened schema structure created by the CSV parser into an R3-style schema.
* [Manifest-based Ingestion](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags) – The manifest ingestion workflow leverages a manifest schema definition defined by the Data Definitions team to facilitate data into the OSDU® Data Platform. - [Manifest-based Ingestion](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags) – The manifest ingestion workflow leverages a manifest schema definition defined by the Data Definitions team to facilitate data into the OSDU® Data Platform.
* [WITSML Parser Ingestion](https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser) – Energistics have created an ingestion workflow capable of parsing WITSML into R3 schema formats. - [WITSML Parser Ingestion](https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser) – Energistics have created an ingestion workflow capable of parsing WITSML into R3 schema formats.
The fundamental idea in each of these ingestion methods is to trigger [storage service API](https://community.opengroup.org/osdu/platform/system/storage) to create the records. Alternatively, one can also run the storage service API directly to create the records but note that this approach is very forgiving and could lead to unexpected behavior. The fundamental idea in each of these ingestion methods is to trigger [storage service API](https://community.opengroup.org/osdu/platform/system/storage) to create the records. Alternatively, one can also run the storage service API directly to create the records but note that this approach is very forgiving and could lead to unexpected behavior.
...@@ -92,20 +92,25 @@ This guide assumes you have access to a working OSDU® environment, please conta ...@@ -92,20 +92,25 @@ This guide assumes you have access to a working OSDU® environment, please conta
In this quickstart guide, we will use the [open-test-data](https://community.opengroup.org/osdu/data/open-test-data) to demonstrate the steps above. In this example, we describe one of the three methods described above - Manifest-based Ingestion. In this quickstart guide, we will use the [open-test-data](https://community.opengroup.org/osdu/data/open-test-data) to demonstrate the steps above. In this example, we describe one of the three methods described above - Manifest-based Ingestion.
* **Manifest-based Ingestion** - **Manifest-based Ingestion**
1. Load reference data in the OSDU® Data Platform 1. Load reference data in the OSDU® Data Platform
* For TNO example, the reference data [manifests](https://community.opengroup.org/osdu/data/open-test-data/-/tree/master/rc--3.0.0/4-instances/TNO/reference-data) should first be loaded into the OSDU® platform.
* Any other missing reference data can be found in the OSDU® community [reference data](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/ReferenceValues/Manifests/reference-data). This repository is maintained by the Data Definitions team. - For TNO example, the reference data [manifests](https://community.opengroup.org/osdu/data/open-test-data/-/tree/master/rc--3.0.0/4-instances/TNO/reference-data) should first be loaded into the OSDU® platform.
- Any other missing reference data can be found in the OSDU® community [reference data](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/ReferenceValues/Manifests/reference-data). This repository is maintained by the Data Definitions team.
2. Prepare the master/WPC data manifests JSON 2. Prepare the master/WPC data manifests JSON
* Here is a set of [Python data preparation scripts](https://community.opengroup.org/osdu/data/open-test-data/-/tree/master/rc--3.0.0/2-scripts) to help with the manifest generation.
* You can either [learn](https://community.opengroup.org/groups/osdu/platform/data-flow/data-loading/-/wikis/How-to-generate-manifests-using-scripts) to generate them from scratch with the [scripts](https://community.opengroup.org/osdu/data/open-test-data/-/tree/master/rc--3.0.0/2-scripts) or use to ones that have been [generated](https://community.opengroup.org/osdu/data/open-test-data/-/tree/master/rc--3.0.0/4-instances) from the scripts. - Here is a set of [Python data preparation scripts](https://community.opengroup.org/osdu/data/open-test-data/-/tree/master/rc--3.0.0/2-scripts) to help with the manifest generation.
- You can either [learn](https://community.opengroup.org/groups/osdu/platform/-/wikis/How-to-generate-manifests-using-scripts) to generate them from scratch with the [scripts](https://community.opengroup.org/osdu/data/open-test-data/-/tree/master/rc--3.0.0/2-scripts) or use to ones that have been [generated](https://community.opengroup.org/osdu/data/open-test-data/-/tree/master/rc--3.0.0/4-instances) from the scripts.
3. Load master/WP/WPC data in the OSDU® Data Platform 3. Load master/WP/WPC data in the OSDU® Data Platform
* Send a POST request to `{OSDU_BASE_URL}/api/workflow/v1/workflow/Osdu_ingest/workflowRun` with the manifest JSON in the request body to trigger the workflow ingestion service as shown in an example below:
- Send a POST request to `{OSDU_BASE_URL}/api/workflow/v1/workflow/Osdu_ingest/workflowRun` with the manifest JSON in the request body to trigger the workflow ingestion service as shown in an example below:
```json ```json
{"executionContext": { {
"executionContext": {
"Payload": { "Payload": {
"AppKey": "test-app", "AppKey": "test-app",
"data-partition-id": "{{data-partition-id}}" "data-partition-id": "{{data-partition-id}}"
...@@ -117,20 +122,12 @@ In this quickstart guide, we will use the [open-test-data](https://community.ope ...@@ -117,20 +122,12 @@ In this quickstart guide, we will use the [open-test-data](https://community.ope
"id": "{{data-partition-id}}:reference-data--FacilityType:Well-{{randomId}}", "id": "{{data-partition-id}}:reference-data--FacilityType:Well-{{randomId}}",
"kind": "{{authority}}:wks:reference-data--FacilityType:1.0.0", "kind": "{{authority}}:wks:reference-data--FacilityType:1.0.0",
"acl": { "acl": {
"owners": [ "owners": ["data.default.owners@{{data-partition-id}}{{domain}}"],
"data.default.owners@{{data-partition-id}}{{domain}}" "viewers": ["data.default.viewers@{{data-partition-id}}{{domain}}"]
],
"viewers": [
"data.default.viewers@{{data-partition-id}}{{domain}}"
]
}, },
"legal": { "legal": {
"legaltags": [ "legaltags": ["{{LegalTagName}}"],
"{{LegalTagName}}" "otherRelevantDataCountries": ["US"]
],
"otherRelevantDataCountries": [
"US"
]
}, },
"data": { "data": {
"Name": "Well", "Name": "Well",
...@@ -140,23 +137,26 @@ In this quickstart guide, we will use the [open-test-data](https://community.ope ...@@ -140,23 +137,26 @@ In this quickstart guide, we will use the [open-test-data](https://community.ope
} }
} }
] ]
}}} }
}
}
``` ```
4. Verify the data has been properly loaded 4. Verify the data has been properly loaded
* Send a GET request to storage records API `{{OSDU_BASE_URL}}/api/storage/v2/records/{{id}}` - Send a GET request to storage records API `{{OSDU_BASE_URL}}/api/storage/v2/records/{{id}}`
* The id is the one you passed in the manifest JSON. It should look something like this: - The id is the one you passed in the manifest JSON. It should look something like this:
```json ```json
{{data-partition-id}}:reference-data--FacilityType:Well-183038 {{data-partition-id}}:reference-data--FacilityType:Well-183038
``` ```
* If the manifest is ingested successfully, you should see this in the response body: - If the manifest is ingested successfully, you should see this in the response body:
```json ```json
{"data": { {
"data": {
"ID": "Well", "ID": "Well",
"Code": "Well", "Code": "Well",
"Source": "Workbook Published/FacilityTypeType.1.0.0.xlsx; commit SHA 0b4db59a.", "Source": "Workbook Published/FacilityTypeType.1.0.0.xlsx; commit SHA 0b4db59a.",
...@@ -167,20 +167,12 @@ In this quickstart guide, we will use the [open-test-data](https://community.ope ...@@ -167,20 +167,12 @@ In this quickstart guide, we will use the [open-test-data](https://community.ope
"version": 1638250858097113, "version": 1638250858097113,
"kind": "osdu:wks:reference-data--FacilityType:1.0.0", "kind": "osdu:wks:reference-data--FacilityType:1.0.0",
"acl": { "acl": {
"viewers": [ "viewers": ["data.default.viewers@opendes.ibm.com"],
"data.default.viewers@opendes.ibm.com" "owners": ["data.default.owners@opendes.ibm.com"]
],
"owners": [
"data.default.owners@opendes.ibm.com"
]
}, },
"legal": { "legal": {
"legaltags": [ "legaltags": ["opendes-Wellbore-Legal-Tag-Test8703354"],
"opendes-Wellbore-Legal-Tag-Test8703354" "otherRelevantDataCountries": ["US"],
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant" "status": "compliant"
}, },
"createUser": "airflow-user@osdu.opengroup.org", "createUser": "airflow-user@osdu.opengroup.org",
...@@ -225,19 +217,19 @@ Here’s an example of a batch mode input format for the workflow run API (simpl ...@@ -225,19 +217,19 @@ Here’s an example of a batch mode input format for the workflow run API (simpl
This section runs through the common tasks in data loading and ingestions. Refer to the links in each section to dive deeper. This section runs through the common tasks in data loading and ingestions. Refer to the links in each section to dive deeper.
1. [How to generate manifests using scripts](https://community.opengroup.org/groups/osdu/platform/-/wikis/How-to-generate-manifests-using-scripts) - by Yanbin Zhang [Chevron] 1. [How to generate manifests using scripts](https://community.opengroup.org/groups/osdu/platform/-/wikis/How-to-generate-manifests-using-scripts) - by Yanbin Zhang [Chevron]
2. [How to load a LAS data (Manifest-based Ingestion, Metadata only, without Wellbore DDMS](https://community.opengroup.org/groups/osdu/platform/data-flow/data-loading/-/wikis/How-to-load-a-LAS-data-(Manifest-based-Ingestion)) - by Ivar Sørheim [Equinor] 2. [How to load a LAS data (Manifest-based Ingestion, Metadata only, without Wellbore DDMS](<https://community.opengroup.org/groups/osdu/platform/-/wikis/How-to-load-a-LAS-data-(Manifest-based-Ingestion)>) - by Ivar Sørheim [Equinor]
3. [How to perform a CSV Ingestion with Dataset service](https://community.opengroup.org/groups/osdu/platform/data-flow/data-loading/-/wikis/CSV-Ingestion) - by Chad Leong [SLB] 3. [How to perform a CSV Ingestion with Dataset service](https://community.opengroup.org/groups/osdu/platform/-/wikis/CSV-Ingestion) - by Chad Leong [SLB]
3. [How to perform a CSV Ingestion with File service](https://community.opengroup.org/groups/osdu/platform/data-flow/data-loading/-/wikis/CSV-Ingestion-File-Service) - by Samiullah Ghousudeen [BP] 4. [How to perform a CSV Ingestion with File service](https://community.opengroup.org/groups/osdu/platform/-/wikis/CSV-Ingestion-File-Service) - by Samiullah Ghousudeen [BP]
4. [How to set up sdutil for uploading seismic data in Windows](https://community.opengroup.org/groups/osdu/platform/data-flow/data-loading/-/wikis/Step-by-Step-of-Setting-up-Python-Environment-for-sdutil-in-Windows) - by Chad Leong [SLB] 5. [How to set up sdutil for uploading seismic data in Windows](https://community.opengroup.org/groups/osdu/platform/-/wikis/Step-by-Step-of-Setting-up-Python-Environment-for-sdutil-in-Windows) - by Chad Leong [SLB]
5. [How to load a seismic data via Seismic DDMS - SEGY to oZGY (Storage service API)](https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/blob/master/doc/testing.md) - by Andras Szalai [EPAM] 6. [How to load a seismic data via Seismic DDMS - SEGY to oZGY (Storage service API)](https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/blob/master/doc/testing.md) - by Andras Szalai [EPAM]
6. [How to load a seismic data via Seismic DDMS - SEGY to oZGY](https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/blob/master/doc/gcp/QUICKSTART.md) - by Yan Sushchynski [EPAM] 7. [How to load a seismic data via Seismic DDMS - SEGY to oZGY](https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/blob/master/doc/gcp/QUICKSTART.md) - by Yan Sushchynski [EPAM]
7. [How to load a seismic data via Seismic DDMS - SEGY to oVDS (Storage service API)](https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-vds-conversion/-/blob/master/docs/gcp/QUICKSTART.md) - by Yan Sushchynski [EPAM] 8. [How to load a seismic data via Seismic DDMS - SEGY to oVDS (Storage service API)](https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-vds-conversion/-/blob/master/docs/gcp/QUICKSTART.md) - by Yan Sushchynski [EPAM]
8. [How to load a WITSML data (WITSML Parser)](https://community.opengroup.org/groups/osdu/platform/data-flow/data-loading/-/wikis/How-to-load-a-WITSML-data-(WITSML-Parser)) - by Kateryna Kurach [EPAM] 9. [How to load a WITSML data (WITSML Parser)](<https://community.opengroup.org/groups/osdu/platform/-/wikis/How-to-load-a-WITSML-data-(WITSML-Parser)>) - by Kateryna Kurach [EPAM]
9. [How to load a generic file in OSDU® using File/Dataset service](https://community.opengroup.org/groups/osdu/platform/data-flow/data-loading/-/wikis/How-to-load-a-generic-file-in-OSDU-using-File-or-Dataset-Service) - by Kateryna Kurach [EPAM] 10. [How to load a generic file in OSDU® using File/Dataset service](https://community.opengroup.org/groups/osdu/platform/-/wikis/How-to-load-a-generic-file-in-OSDU-using-File-or-Dataset-Service) - by Kateryna Kurach [EPAM]
9. [How to check for error in Airflow Dag](https://community.opengroup.org/groups/osdu/platform/data-flow/data-loading/-/wikis/How-to-check-for-error-in-Airflow-DAG) - by Chad Leong [SLB] 11. [How to check for error in Airflow Dag](https://community.opengroup.org/groups/osdu/platform/-/wikis/How-to-check-for-error-in-Airflow-DAG) - by Chad Leong [SLB]
10. [How to search for ingested record](https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/docs/tutorial/SearchService.md) 12. [How to search for ingested record](https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/docs/tutorial/SearchService.md)
11. [Troubleshooting Index Status of Data Ingested](https://community.opengroup.org/groups/osdu/platform/data-flow/data-loading/-/wikis/Troubleshooting-Index-Status-of-Data-Ingested) - by Samiullah Ghousudeen [BP] 13. [Troubleshooting Index Status of Data Ingested](https://community.opengroup.org/groups/osdu/platform/-/wikis/Troubleshooting-Index-Status-of-Data-Ingested) - by Samiullah Ghousudeen [BP]
12. [Wellbore DDMS Data Loader Utility Quickstart guide](https://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/wikis/Wellbore-DDMS-Data-Loader-Utility-Quickstart-Guide) - by Samiullah Ghosudeen [BP] 14. [Wellbore DDMS Data Loader Utility Quickstart guide](https://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/wikis/Wellbore-DDMS-Data-Loader-Utility-Quickstart-Guide) - by Samiullah Ghosudeen [BP]
## Bulk loading ## Bulk loading
...@@ -256,35 +248,35 @@ Here are some [worked examples](https://community.opengroup.org/osdu/data/data-d ...@@ -256,35 +248,35 @@ Here are some [worked examples](https://community.opengroup.org/osdu/data/data-d
5. [DataQuality2](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/DataQuality2) 5. [DataQuality2](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/DataQuality2)
6. [ExternalDataServices](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/ExternalDataServices) 6. [ExternalDataServices](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/ExternalDataServices)
7. [FormationIntegrityTest](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/FormationIntegrityTest)8. [FrameOfReference](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/FrameOfReference) 7. [FormationIntegrityTest](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/FormationIntegrityTest)8. [FrameOfReference](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/FrameOfReference)
9. [GenericGeometries](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/GenericGeometries) 8. [GenericGeometries](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/GenericGeometries)
10. [GeoJSON-properties](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/GeoJSON-properties) 9. [GeoJSON-properties](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/GeoJSON-properties)
11. [GeoReferencedImage](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/GeoReferencedImage) 10. [GeoReferencedImage](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/GeoReferencedImage)
12. [Geomatics](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/Geomatics) 11. [Geomatics](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/Geomatics)
13. [GravityMagnetic/AcquisitionSurvey](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/GravityMagnetic) 12. [GravityMagnetic/AcquisitionSurvey](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/GravityMagnetic)
14. [IndexExtensions](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/IndexExtensions) 13. [IndexExtensions](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/IndexExtensions)
15. [Organisation](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/Organisation) 14. [Organisation](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/Organisation)
16. [ReferenceValueMappings](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/ReferenceValueMappings) 15. [ReferenceValueMappings](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/ReferenceValueMappings)
17. [Reservoir Data](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/Reservoir%20Data) 16. [Reservoir Data](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/Reservoir%20Data)
18. [ReservoirManagement](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/ReservoirManagement) 17. [ReservoirManagement](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/ReservoirManagement)
19. [RockImage](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/RockImage) 18. [RockImage](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/RockImage)
20. [RockSampleAnalysis](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/RockSampleAnalysis) 19. [RockSampleAnalysis](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/RockSampleAnalysis)
21. [Seismic2DLineNavigation](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/Seismic2DLineNavigation) 20. [Seismic2DLineNavigation](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/Seismic2DLineNavigation)
22. [Seismic4D](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/Seismic4D) 21. [Seismic4D](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/Seismic4D)
23. [SeismicLoadingManifests](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/SeismicLoadingManifests) 22. [SeismicLoadingManifests](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/SeismicLoadingManifests)
24. [SeismicPreStack](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/SeismicPreStack) 23. [SeismicPreStack](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/SeismicPreStack)
25. [SeismicWellTie](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/SeismicWellTie) 24. [SeismicWellTie](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/SeismicWellTie)
26. [UnitOfMeasureConfiguration](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/UnitOfMeasureConfiguration) 25. [UnitOfMeasureConfiguration](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/UnitOfMeasureConfiguration)
27. [UnitOfMeasureLookUp](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/UnitOfMeasureLookUp) 26. [UnitOfMeasureLookUp](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/UnitOfMeasureLookUp)
28. [Upgrades](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/Upgrades) 27. [Upgrades](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/Upgrades)
29. [VerticalReferenceUsage](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/VerticalReferenceUsage) 28. [VerticalReferenceUsage](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/VerticalReferenceUsage)
30. [WellCompletions](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/WellCompletions) 29. [WellCompletions](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/WellCompletions)
31. [WellExecution](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/WellExecution) 30. [WellExecution](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/WellExecution)
32. [WellLog](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/WellLog) 31. [WellLog](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/WellLog)
33. [WellPlanning](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/WellPlanning) 32. [WellPlanning](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/WellPlanning)
34. [WellStatusClassification](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/WellStatusClassification) 33. [WellStatusClassification](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/WellStatusClassification)
35. [WellboreMarkerSet](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/WellboreMarkerSet) 34. [WellboreMarkerSet](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/WellboreMarkerSet)
36. [WellboreTrajectory](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/WellboreTrajectory) 35. [WellboreTrajectory](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/WellboreTrajectory)
37. [_Guide](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/_Guide) 36. [\_Guide](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/Examples/WorkedExamples/_Guide)
## Domain Data Management Services ## Domain Data Management Services
...@@ -339,14 +331,14 @@ In the **process_manifest_task** operator, you will see the success log below: ...@@ -339,14 +331,14 @@ In the **process_manifest_task** operator, you will see the success log below:
] ]
[2021-07-05 22:52:32,204] {taskinstance.py:1057} INFO - Marking task as SUCCESS.dag_id=Osdu_ingest, task_id=process_single_manifest_file_task, execution_date=20210705T225116, start_date=20210705T225219, end_date=20210705T225232 [2021-07-05 22:52:32,204] {taskinstance.py:1057} INFO - Marking task as SUCCESS.dag_id=Osdu_ingest, task_id=process_single_manifest_file_task, execution_date=20210705T225116, start_date=20210705T225219, end_date=20210705T225232
``` ```
And in the Xcom summary, you should see the newly created record_id. And in the Xcom summary, you should see the newly created record_id.
| Key | Value | | Key | Value |
| ------ | ------ | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| record_ids | ['osdu:reference-data--GeoPoliticalEntityType:Parish','osdu:reference-data--GeoPoliticalEntityType:Community','osdu:reference-data--GeoPoliticalEntityType:Prefecture', 'osdu:reference-data--GeoPoliticalEntityType:Principality', 'osdu:reference-data--GeoPoliticalEntityType:Province'] | | record_ids | ['osdu:reference-data--GeoPoliticalEntityType:Parish','osdu:reference-data--GeoPoliticalEntityType:Community','osdu:reference-data--GeoPoliticalEntityType:Prefecture', 'osdu:reference-data--GeoPoliticalEntityType:Principality', 'osdu:reference-data--GeoPoliticalEntityType:Province'] |
| skipped_ids | [] | | skipped_ids | [] |
### Invalid ACL - error: ### Invalid ACL - error:
If you entered incorrect ACL information, in the **process_manifest_task** operator, you will see the following error log below: If you entered incorrect ACL information, in the **process_manifest_task** operator, you will see the following error log below:
...@@ -359,7 +351,7 @@ If you entered incorrect ACL information, in the **process_manifest_task** opera ...@@ -359,7 +351,7 @@ If you entered incorrect ACL information, in the **process_manifest_task** opera
``` ```
| Key | Value | | Key | Value |
|-------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| record_ids | [] | | record_ids | [] |
| skipped_ids | [{'id': 'osdu:master-data--Organisation:SLB', 'kind': 'osdu:wks:master-data--Organisation:1.0.0', 'reason': '400 Client Error: Bad Request for url: http://os-storage.osdu-services:8080/api/storage/v2/records'}] | | skipped_ids | [{'id': 'osdu:master-data--Organisation:SLB', 'kind': 'osdu:wks:master-data--Organisation:1.0.0', 'reason': '400 Client Error: Bad Request for url: http://os-storage.osdu-services:8080/api/storage/v2/records'}] |
...@@ -371,15 +363,15 @@ Contents of the guide are contributed by different forum members. ...@@ -371,15 +363,15 @@ Contents of the guide are contributed by different forum members.
Contributors: Contributors:
* Chad Leong \[SLB\] - Chad Leong \[SLB\]
* Ivar Sorheim \[Equinor\] - Ivar Sorheim \[Equinor\]
* Debasis Chatterjee \[Katalyst\] - Debasis Chatterjee \[Katalyst\]
* Kateryna Kurach \[EPAM\] - Kateryna Kurach \[EPAM\]
* Keith Wall \[Shell\] - Keith Wall \[Shell\]
* Dadong Zhou \[Shell\] - Dadong Zhou \[Shell\]
* Ash Sathyaseelan \[Infosys\] - Ash Sathyaseelan \[Infosys\]
* Alan Henson \[Pariveda\] - Alan Henson \[Pariveda\]
* Marcus Apel \[Equinor\] - Marcus Apel \[Equinor\]
* Thomas Gehrmann \[SLB\]... - Thomas Gehrmann \[SLB\]...
_More credits to be updated_ _More credits to be updated_