QUICKSTART.md

Quick Start instructions for work with SEGY -> OpenVDS parser (Pre-ship)

• Quick Start instructions for work with SEGY -> OpenVDS parser (Pre-ship)
  • Prerequisites
    • Required Entitlements groups
    • Generate ID token
    • Install SDUTIL
    • Other issues
  • Segy -> OpenVDS conversion
    • Create Seismic Store's tenant and subprojects
    • Upload Segy-file from you machine USING SDUTIL
    • Start Segy -> OpenVDS conversion workflow

This guide is about how to run Segy->OpenVDS conversion Workflow on Pre-ship.

Prerequisites

Required Entitlements groups

Be sure you are in the following Entitlements groups(GET: {{entitlements_api_url}}/entitlements/v2/groups):

• users.datalake.admins
• app.trusted
• service.seistore.admin
• service.entitlements.user
• service.workflow.creator
• seistore.system.admin

Generate ID token

Seismic Store requires well-generated ID token.

Follow this guide to get API Credentials: https://community.opengroup.org/osdu/documentation/-/wikis/Releases/R2.0/GCP/GCP-Operation/User-Mng/OpenID-Connect#how-to-get-api-credentials-using-gcp-openid-playgroud

The most common issue with generating right ID token is using wrong ClientID and ClientSecrets. Contact our devopses to obtain the right ones.

Install SDUTIL

SDUTIL is a command line Python utility designed to work easily with Seismic Store.

Follow the installation guide here: https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil#installation

Replace the content of seismic-store-sdutil/sdlib/config.yaml with this:

seistore:
  service: '{"google": {"defaultEnv":{"url": "<seismic_store_host>/api/v3", "appkey": ""}}}'
  url: '<seismic_store_host>/api/v3'
  cloud_provider: 'google'
  key: ''
  env: 'defaultEnv'
  auth-mode: 'JWT Token'
  ssl_verify: False
  APPKEY: ''
  APPKEY_NAME: ''
auth_provider:
  default: ''
gcp:
  empty: 'none'

Then, initialize configs:

python sdutil config init

And choose the first option:

[1] google

Select the cloud provider: 1

Then, you may skip inserting the google (defaultEnv) application key.

Now your SDUTIL is ready to use with JWT token (your ID token).

Example:

python sdutil stat sd://<your_tenant> --idtoken=$ID_token

Other issues

You may find it useful to read this page: https://community.opengroup.org/osdu/documentation/-/wikis/Releases/R2.0/GCP/GCP-Pre-Prod-Onboarding-Documentation

Segy -> OpenVDS conversion

Create Seismic Store's tenant and subprojects

The seismic store uri is a string used to uniquely address a resource in the system and can be obtained by appending the prefix sd:// before the required resource path: sd://<tenant>/<subproject>/<path>*/<dataset>

Before you start uploading the file, you may create tenant and subproject.

If you want to use already created tenant and subproject, ask the owner (creator) of the subroject to add you to it:

curl --location --request PUT '<seismic_store_host>/api/v3/user' \
--header 'Authorization: Bearer <id_token>' \
--header 'Content-Type: application/json' \
--header 'x-api-key: test' \
--header 'appkey: test' \
--data-raw '{
	"email": "<your_email>@<domain>.com",
	"path": "sd://<tenant>/<subproject>",
	"group": "editor"
}'

This command will add you to required Entitlements groups to work with the concrete subproject.

Create the tenant:

curl --location -g --request POST '<seismic_store_host>/api/v3/tenant/<new-tenant>' \
--header 'Authorization: Bearer <id_token>' \
--header 'data-partition-id: <data-partition-id>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "gcpid": "<pre-ship GCP project ID>",
    "esd": "<data-partition-id>.<domain>.com",
    "default_acl": "data.default.owners@<data-partition-id>.<domain>.com"
}'

Then, create the subproject:

curl --location -g --request POST '<seismic_store_host>/api/v3/subproject/tenant/<new-tenant>/subproject/<subproject>' \
--header 'Authorization: Bearer <id_token>' \
--header 'Content-Type: application/json' \
--header 'ltag: <data-partition-id>-demo-legaltag' \
--header 'appkey: test' \
--header 'data-partition-id: <data-partition-id>' \
--data-raw '{

    "storage_class": "MULTI_REGIONAL",
    "storage_location": "US",
    "acl": {
        "owners": [
            "data.default.owners@<data-partition-id>.<domain>.com"
        ],
        "viewers": [
            "data.default.viewers@<data-partition-id>.<domain>.com"
        ]
    }
}'

Upload Segy-file from you machine using SDUTIL

After you installed SDUTIL you must have:

• all required groups in Entitlements;
• well-generated ID token;
• properly configured SDUTIL;
• created Seismic Store's tenant and subproject with access to it.

Copy the file from your local machine to Seismic Store:

python sdutil cp /path/to/local/segy/file sd://<tenant>/<subproject>/<some_path>/<file_name> --idtoken=$ID_TOKEN

From this point your Segy-file is available as a dataset in Seismic Store.

To get the dataset's short info:

python sdutil stat sd://<tenant>/<subproject>/<some_path>/<file_name> --idtoken=$ID_TOKEN

Sometimes it is required to unlock your uploaded dataset

python sdutil unlock sd://<tenant>/<subproject>/<path>/<file_name>  --idtoken=$ID_TOKEN

sd://<tenant>/<subproject>/<some_path>/<file_name> is URI to address your dataset. With this URI you will work in the next steps.

Ingest the WorkProduct of the Segy-file

Ingest the corresponding Manifest with using Manifest-based-ingestion. The sd-path must be set as a value of data.DatasetsProperties.FileCollectionPath of the dataset--FileCollection.SEGY record.

Then, you can use the Ids of the File and WorkProduct records for the further conversion.

Start Segy -> OpenVDS conversion workflow

After you uploaded the file on Seismic Store and created the metadata of the file, you can start the conversion workflow:

curl --location --request POST '<workflow_host>/v1/workflow/Segy_to_vds_conversion_sdms/workflowRun' \
--header 'data-partition-id: <data-partition-id>' \
--header 'Authorization: Bearer <access_token>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "executionContext": {
        "Payload": {
            "AppKey": "test-app",
            "data-partition-id": "<data-partition-id>"
        },
        "vds_url": "sd://<tenant>/<subproject>/<path>",
        "persistent_id": "<unique name of vds conversion>",
        "id_token": <id_token>
        "work_product_id": "<work-product-id>"
        "file_record_id": <vds-file-record-id>
    }

After the conversion, a new OpenVDS FileRecord will be created with the sd-path to the OpenVDS collection in it. Also, the SeismicTraceData record will be updated with Artefacts field with the reference to the OpenVDS file.

The following fields:

• vds_url - the part of the OpenVDS dataset Seismic Store URI consisting of tenant, subproject, and path;

• persistent_id - unique ID of the dataset, can be considered the dataset's name;

• file_record_id - Segy-file metadata with Seismic Store URI.

• work_product_id - Work product id with WPC that have references to the FIle Record

The full Seismic Store URI of OpenVDS dataset will look like sd://<tenant>/<subproject>/<path>/<persistent_id>.

To verify that OpenVDS collection was created successfully:

python sdutil stat sd://<tenant>/<subproject>/<path>/<persistent_id> --idtoken=$ID_TOKEN

or

curl --location --request GET '<seismic_store_host>/api/v3/dataset/tenant/<tenant>/subproject/<subproject>/dataset/<persistent_id>?path=<path>&seismicmeta=true' \
--header 'Authorization: Bearer <id_token>' \
--header 'data-partition-id: <data-partition-id>'