Introduction

Wellbore Data Management Services (WDMS) Open Subsurface Data Universe (OSDU) is one of the several backend services that comprise Schlumberger's Exploration and Production (E&P) software ecosystem. It is a single, containerized service written in Python that provides an API for wellbore related data.

• Introduction
  • Install Software and Packages
    • Fast API Dependencies
    • Additional Dependencies
    • Library Dependencies
  • Project Startup
    • Run the service locally
    • Connect and Run Endpoints
    • Setting the Cloud Provider Environment Variables
    • Create a log record
    • Run with Uvicorn
    • Run with Docker
      • Build Image
      • Run Image
    • Run Unit Tests Locally
    • Port Forward from Kubernetes
    • Tracing

Install Software and Packages

1. Clone the os-wellbore-ddms repository

2. Download Python >=3.7

3. Ensure pip, a pre-installed package manager and installer for Python, is installed and is upgraded to the latest version.

   # Windows
   python -m pip install --upgrade pip
   python -m pip --version
   
   # macOS and Linux
   python3 -m pip install --upgrade pip
   python3 -m pip --version

4. Using pip, download FastAPI, the main framework to build the service APIs. To install fastapi and uvicorn (to work as the server), run the following command:

   pip install fastapi[all]

5. venv allows you to manage separate package installations for different projects. They essentially allow you to create a "virtual" isolated Python installation and packages into that virtual environment. venv is already included in the Python standard library and requires no additional installation.

Fast API Dependencies

• pydantic: provides the ability to do data validation using python type annotations. It enforces type hints at runtime provide a more robust data validation option.
  • dataclasses: module in python which provides a decorator and functions for automatically adding generated special methods to user-defined classes.
• starlette: lightweight ASGI framework. FastAPI is a sub-class of Starlette and includes features such as websocket support, startup and shutdown events, session and cookie support.

Additional Dependencies

• uvicorn used as ASGI server to run WDMS app
• cachetools
• pyjwt and cryptography for auth purposes
• pandas and numpy for data manipulation
• pyarrow for load and save data into parquet format
• opencensus for tracing and logging on cloud provider

Library Dependencies

• Common parts and interfaces

  • osdu-core-python

• Implementation of blob storage on GCP

  • osdu-core-python-gcp

• Implementation of blob storage and partition service on Azure

  • osdu-core-python-azure

• Storage, search and entitlements

  • osdu-python-clients

Project Startup

Run the service locally

1. Create virtual environment in the wellbore project directory. This will create a folder inside of the wellbore project directory. For example: ~/os-wellbore-ddms/nameofvirtualenv

   # Windows
   python -m venv env
   
   # macOS/Linux
   python3 -m venv env

2. Activate the virtual environment

   # Windows
   source env/Scripts/activate
   
   # macOS/Linux
   source env/bin/activate

3. Create pip.ini (Windows) or pip.conf (MacOS and Linux) file inside the env directory. This allows us to set a global index url which can download packages/libraries needed from the AzDO artifacts. There are several ways to add this extra index url:

   • It is also possible to use --extra-index-url parameter to specify it on the pip install cmd inline

4. Install dependencies

   pip install -r requirements.txt

5. Run the service

   # Run the service which will default to http://127.0.0.1:8080
   python main.py
   
   # Run on specific host, port and enforce dev mode
   python main.py --host MY_HOST --port MY_PORT --dev_mode 1

   If host is 127.0.0.1 or localhost, the dev_mode is automatically set to True. The only significant change if dev_mode is on, is that configuration errors at startup are logged but don’t prevent the service to run, and allow to override some implementations.

The hosts for the entitlements, search and storage services have to be provided as environment variables, or on the command line.

python main.py -e SERVICE_HOST_ENTITLEMENTS https://api.example.com/entitlements -e SERVICE_HOST_STORAGE https://api.example.com/storage -e SERVICE_HOST_SEARCH https://api.example.com/search

Connect and Run Endpoints

1. Generate bearer token as all APIs but /about require authentication.

   • Navigate to http://127.0.0.1:8080/token and follow the steps to generate a bearer token.

   • Navigate to http://127.0.0.1:8080/docs. Click Authorize and enter your token. That will allow for authenticated requests.

2. Choose storage option

   Even if the service runs locally it still relies on osdu data ecosystem storage service os-storage-dot-opendes.appspot.com/api/storage to store documents and google blob store to store binary data (bulk data). It is possible to override this and use your local file system instead by setting the following environment variables:

   • USE_INTERNAL_STORAGE_SERVICE_WITH_PATH to store on a local folder instead of osdu ecosystem storage service.
   • USE_LOCALFS_BLOB_STORAGE_WITH_PATH to store on a local folder instead of google blob storage.

   # Create temp storage folders
   mkdir tmpstorage
   mkdir tmpblob
   
   # Set your repo path
   path="C:/source"
   
   python main.py -e USE_INTERNAL_STORAGE_SERVICE_WITH_PATH $path/os-wellbore-ddms/tmpstorage -e USE_LOCALFS_BLOB_STORAGE_WITH_PATH $path/os-wellbore-ddms/tmpblob

3. Choose Cloud Provider

   • The code can be run with specifying environment variables and by setting the cloud provider. The accepted values are gcp, az or local. When a cloud provider is passed as an environment variables, there are certain additional environment variables that become mandatory.

Setting the Cloud Provider Environment Variables

• The following environment variables are required when the cloud provider is set to GCP:

  • OS_WELLBORE_DDMS_DATA_PROJECT_ID: GCP Data Tenant ID
  • OS_WELLBORE_DDMS_DATA_PROJECT_CREDENTIALS: path to the key file of the SA to access the data tenant
  • SERVICE_HOST_ENTITLEMENTS: The Entitlements Service host
  • SERVICE_HOST_SEARCH: The Search Service host
  • SERVICE_HOST_STORAGE: The Storage Service host

  python main.py -e CLOUD_PROVIDER gcp \
  -e OS_WELLBORE_DDMS_DATA_PROJECT_ID projectid \
  -e OS_WELLBORE_DDMS_DATA_PROJECT_CREDENTIALS pathtokeyfile \
  -e SERVICE_HOST_ENTITLEMENTS entitlement_host \
  -e SERVICE_HOST_SEARCH search_host \
  -e SERVICE_HOST_STORAGE storage_host

• The following environment variables are required when the cloud provider is set to Azure:

  • AZ_AI_INSTRUMENTATION_KEY: Azure Application Insights instrumentation key
  • SERVICE_HOST_ENTITLEMENTS: The Entitlements Service host
  • SERVICE_HOST_SEARCH: The Search Service host
  • SERVICE_HOST_STORAGE: The Storage Service host
  • SERVICE_HOST_PARTITION: The Partition Service internal host
  • KEYVAULT_URL: The Key Vault url (needed by the Partition Service)
  • USE_PARTITION_SERVICE: enabled when Partition Service is available in the environment. Needs to be disabled for dev or to run locally.

  python main.py -e CLOUD_PROVIDER az \
  -e AZ_AI_INSTRUMENTATION_KEY instrumentationkey \
  -e SERVICE_HOST_ENTITLEMENTS entitlement_host \
  -e SERVICE_HOST_SEARCH search_host \
  -e SERVICE_HOST_STORAGE storage_host \
  -e SERVICE_HOST_PARTITION partition_host \
  -e KEYVAULT_URL keyvault_url \
  -e USE_PARTITION_SERVICE disabled

Note: If you're running locally, you may need to provide environmental variables in your IDE. Here is a sample for providing a .env file.

As default, all Core Services endpoint values are set to None in app/conf.py, you can update .env file for core services endpoints based on your cloud provider.

Create a log record

To create a log record, below is a payload sample for the PUT /ddms/v2/logs API. The response will contain an id you can use on the /ddms/v2/logs/{logid}/data to create some bulk data.

• GCP

  [{
          "data": {
              "log": {
                  "family": "Gamma Ray",
                  "familyType": "Gamma Ray",
                  "format": "float64",
                  "mnemonic": "GR",
                  "name": "GAMM",
                  "unitKey": "gAPI"
              }
          },
          "kind": "opendes:osdu:log:1.0.5",
          "namespace": "opendes:osdu",
          "legal": {
              "legaltags": [
                  "opendes-public-usa-dataset-1"
              ],
              "otherRelevantDataCountries": [
                  "US"
              ],
              "status": "compliant"
          },
          "acl": {
              "viewers": [
                  "data.default.viewers@opendes.p4d.cloud.slb-ds.com"
              ],
              "owners": [
                  "data.default.owners@opendes.p4d.cloud.slb-ds.com"
              ]
          },
          "type": "log"
      }
  ]

• MVP

  [
      {
          "acl": {
              "owners": [
                  "data.default.owners@opendes.contoso.com"
              ],
              "viewers": [
                  "data.default.viewers@opendes.contoso.com"
              ]
          },
          "data": {
              "name": "wdms_e2e_log"
          },
          "kind": "opendes:wks:log:1.0.5",
          "legal": {
              "legaltags": [
                  "opendes-storage-1603197111615"
              ],
              "otherRelevantDataCountries": [
                  "US",
                  "FR"
              ]
          }
      }
  ]

Run with Uvicorn

uvicorn app.wdms_app:wdms_app --port LOCAL_PORT

Then access app on http://localhost:LOCAL_PORT/docs

Run with Docker

Build Image

A Personal Access Token (PAT) is required to pull all the python packages.

USER=<username>
PAT=<PAT>

# Set PIP_EXTRA_URL
PIP_EXTRA_URL="<URL>"

# Set IMAGE_TAG
IMAGE_TAG="os-wellbore-ddms:dev"

# Build Image
docker build -t=$IMAGE_TAG --rm . -f ./build/dockerfile --build-arg PIP_EXTRA_URL=$PIP_EXTRA_URL --build-arg PIP_WHEEL_DIR=python-packages

Run Image

1. Run the image

   Replace the LOCAL_PORT value with a local port

   LOCAL_PORT=<local_port>
   
   docker run -d -p $LOCAL_PORT:8080 -e OS_WELLBORE_DDMS_DEV_MODE=1 -e USE_LOCALFS_BLOB_STORAGE_WITH_PATH=1 $IMAGE_TAG

2. Access app on http://localhost:LOCAL_PORT/docs

3. The environment variable OS_WELLBORE_DDMS_DEV_MODE=1 enables dev mode

4. Logs can be checked by running

   docker logs CONTAINER_ID

Run Unit Tests Locally

# Install test dependencies
pip install -r requirements_dev.txt

python -m pytest --junit-xml=unit_tests_report.xml --cov=app --cov-report=html --cov-report=xml ./tests/unit

Coverage reports can be viewed after the command is run. The HMTL reports are saved in the htmlcov directory.

Port Forward from Kubernetes

1. List the pods: kubectl get pods
2. Port forward: kubectl port-forward pods/POD_NAME LOCAL_PORT:8080
3. Access it on http://localhost:LOCAL_PORT/docs

Tracing

OpenCensus libraries are used to record incoming requests metrics (execution time, result code, etc...). At the moment, 100% of the requests are saved.