README.md 13.3 KB
Newer Older
1
2
# Introduction

Luc Yriarte's avatar
Luc Yriarte committed
3
Wellbore Domain Data Management Services (Wellbore-DDMS) Open Subsurface Data Universe (OSDU) is one of the several backend services that comprise OSDU software ecosystem. It is a single, containerized service written in Python that provides an API for wellbore related data.
4

Luc Yriarte's avatar
Luc Yriarte committed
5
6
[[_TOC_]]

7
8
## Install Software and Packages

Luc Yriarte's avatar
Luc Yriarte committed
9
1. Clone the os-wellbore-ddms [repository](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services.git)
10
11
12
2. Download [Python](https://www.python.org/downloads/) >=3.7
3. Ensure pip, a pre-installed package manager and installer for Python, is installed and is upgraded to the latest version.

Luc Yriarte's avatar
Luc Yriarte committed
13
14
15
16
      ```bash
      # Windows
      python -m pip install --upgrade pip
      python -m pip --version
17

Luc Yriarte's avatar
Luc Yriarte committed
18
19
20
21
      # macOS and Linux
      python3 -m pip install --upgrade pip
      python3 -m pip --version
      ```
22
23
24

4. Using pip, download [FastAPI](https://fastapi.tiangolo.com/), the main framework to build the service APIs. To install fastapi and uvicorn (to work as the server), run the following command:

Luc Yriarte's avatar
Luc Yriarte committed
25
26
27
    ```bash
    pip install fastapi[all]
    ```
28

Luc Yriarte's avatar
Luc Yriarte committed
29
5. [venv](https://docs.python.org/3/library/venv.html) allows you to manage separate package installations for different projects. They essentially allow you to create a "virtual" isolated Python installation and packages into that virtual environment. venv is already included in the Python standard library and requires no additional installation.
30
31
32
33
34
35
36
37
38

### Fast API Dependencies

- [pydantic](https://pydantic-docs.helpmanual.io/): provides the ability to do data validation using python type annotations. It enforces type hints at runtime provide a more robust data validation option.
  - [dataclasses](https://pydantic-docs.helpmanual.io/usage/dataclasses/): module in python which provides a decorator and functions for automatically adding generated special methods to user-defined classes.
- [starlette](https://fastapi.tiangolo.com/features/#starlette-features): lightweight ASGI framework. FastAPI is a sub-class of Starlette and includes features such as websocket support, startup and shutdown events, session and cookie support.

### Additional Dependencies

39
- [uvicorn](https://www.uvicorn.org/) used as ASGI server to run Wellbore-DDMS app
40
41
42
43
44
45
46
47
48
- [cachetools](https://pypi.org/project/cachetools/)
- [pyjwt](https://pypi.org/project/PyJWT/) and [cryptography](https://pypi.org/project/cryptography/) for auth purposes
- [pandas](https://pandas.pydata.org/) and [numpy](https://numpy.org/) for data manipulation
- [pyarrow](https://pypi.org/project/pyarrow/) for load and save data into parquet format
- [opencensus](https://opencensus.io/guides/grpc/python/) for tracing and logging on cloud provider

### Library Dependencies

- Common parts and interfaces
49
  - osdu-core-lib-python
50
51

- Implementation of blob storage on GCP
52
  - osdu-core-lib-python-gcp
53

Luc Yriarte's avatar
Luc Yriarte committed
54
- Implementation of blob storage and partition service on Azure
55
  - osdu-core-lib-python-azure
Luc Yriarte's avatar
Luc Yriarte committed
56

57
58
59
- Client libraries for OSDU data ecosystem services
  - osdu-data-ecosystem-search
  - osdu-data-ecosystem-storage
60
61
62
63
64

## Project Startup

### Run the service locally

Luc Yriarte's avatar
Luc Yriarte committed
65
1. Create virtual environment in the wellbore project directory. This will create a folder inside of the wellbore project directory. For example: ~/os-wellbore-ddms/nameofvirtualenv
66

Luc Yriarte's avatar
Luc Yriarte committed
67
68
69
    ```bash
    # Windows
    python -m venv env
70

Luc Yriarte's avatar
Luc Yriarte committed
71
72
73
    # macOS/Linux
    python3 -m venv env
    ```
74
75
76

2. Activate the virtual environment

Luc Yriarte's avatar
Luc Yriarte committed
77
78
79
    ```bash
    # Windows
    source env/Scripts/activate
80

Luc Yriarte's avatar
Luc Yriarte committed
81
82
83
    # macOS/Linux
    source env/bin/activate
    ```
84

Luc Yriarte's avatar
Luc Yriarte committed
85
5. Install dependencies
86

Luc Yriarte's avatar
Luc Yriarte committed
87
88
89
    ```bash
    pip install -r requirements.txt
    ```
90

Luc Yriarte's avatar
Luc Yriarte committed
91
6. Run the service
92

Luc Yriarte's avatar
Luc Yriarte committed
93
94
95
    ```bash
    # Run the service which will default to http://127.0.0.1:8080
    python main.py
96

Luc Yriarte's avatar
Luc Yriarte committed
97
98
99
    # Run on specific host, port and enforce dev mode
    python main.py --host MY_HOST --port MY_PORT --dev_mode 1
    ```
100

Luc Yriarte's avatar
Luc Yriarte committed
101
102
    If host is `127.0.0.1` or `localhost`, the dev_mode is automatically set to True.
    The only significant change if dev_mode is on, is that configuration errors at startup are logged but don’t prevent the service to run, and allow to override some implementations.
103

Luc Yriarte's avatar
Luc Yriarte committed
104
The hosts for the search and storage services have to be provided as environment variables, or on the command line.
105
106

```bash
Luc Yriarte's avatar
Luc Yriarte committed
107
python main.py -e SERVICE_HOST_STORAGE https://api.example.com/storage -e SERVICE_HOST_SEARCH https://api.example.com/search
108
109
110
111
112
113
```

### Connect and Run Endpoints

1. Generate bearer token as all APIs but `/about` require authentication.

114
    - Navigate to `http://127.0.0.1:8080/api/os-wellbore-ddms/docs`. Click `Authorize` and enter your token. That will allow for authenticated requests.
115
116
117
118


2. Choose storage option

Luc Yriarte's avatar
Luc Yriarte committed
119
    Even if the service runs locally it still relies on osdu data ecosystem storage service `os-storage-dot-opendes.appspot.com/api/storage` to store documents and google blob store to store binary data (`bulk data`). It is possible to override this and use your local file system instead by setting the following environment variables:
120

Luc Yriarte's avatar
Luc Yriarte committed
121
122
    - `USE_INTERNAL_STORAGE_SERVICE_WITH_PATH` to store on a local folder instead of osdu ecosystem storage service.
    - `USE_LOCALFS_BLOB_STORAGE_WITH_PATH` to store on a local folder instead of google blob storage.
123

Luc Yriarte's avatar
Luc Yriarte committed
124
125
126
127
    ```bash
    # Create temp storage folders
    mkdir tmpstorage
    mkdir tmpblob
128

Luc Yriarte's avatar
Luc Yriarte committed
129
130
    # Set your repo path
    path="C:/source"
131

Luc Yriarte's avatar
Luc Yriarte committed
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
    python main.py -e USE_INTERNAL_STORAGE_SERVICE_WITH_PATH $path/os-wellbore-ddms/tmpstorage -e USE_LOCALFS_BLOB_STORAGE_WITH_PATH $path/os-wellbore-ddms/tmpblob
    ```

3. Choose Cloud Provider

    - The code can be run with specifying environment variables and by setting the cloud provider. The accepted values are `gcp`, `az` or `local`. When a cloud provider is passed as an environment variables, there are certain additional environment variables that become mandatory.

### Setting the Cloud Provider Environment Variables

- The following environment variables are required when the cloud provider is set to GCP:
  - OS_WELLBORE_DDMS_DATA_PROJECT_ID: GCP Data Tenant ID
  - OS_WELLBORE_DDMS_DATA_PROJECT_CREDENTIALS: path to the key file of the SA to access the data tenant
  - SERVICE_HOST_SEARCH: The Search Service host
  - SERVICE_HOST_STORAGE: The Storage Service host

  ```bash
  python main.py -e CLOUD_PROVIDER gcp \
  -e OS_WELLBORE_DDMS_DATA_PROJECT_ID projectid \
  -e OS_WELLBORE_DDMS_DATA_PROJECT_CREDENTIALS pathtokeyfile \
  -e SERVICE_HOST_SEARCH search_host \
  -e SERVICE_HOST_STORAGE storage_host
  ```

- The following environment variables are required when the cloud provider is set to Azure:
  - AZ_AI_INSTRUMENTATION_KEY: Azure Application Insights instrumentation key
  - SERVICE_HOST_SEARCH: The Search Service host
  - SERVICE_HOST_STORAGE: The Storage Service host
  - SERVICE_HOST_PARTITION: The Partition Service internal host
  - KEYVAULT_URL: The Key Vault url (needed by the Partition Service)
  - USE_PARTITION_SERVICE: `enabled` when Partition Service is available in the environment. Needs to be `disabled` for `dev` or to run locally.

  ```bash
  python main.py -e CLOUD_PROVIDER az \
  -e AZ_AI_INSTRUMENTATION_KEY instrumentationkey \
  -e SERVICE_HOST_SEARCH search_host \
  -e SERVICE_HOST_STORAGE storage_host \
  -e SERVICE_HOST_PARTITION partition_host \
  -e KEYVAULT_URL keyvault_url \
  -e USE_PARTITION_SERVICE disabled
  ```

Note: If you're running locally, you may need to provide environmental variables in your IDE. Here is a sample for providing a `.env` file.

As default, all Core Services endpoint values are set to `None` in `app/conf.py`, you can update `.env` file for core services endpoints based on your cloud provider.

### Create a log record
178

Luc Yriarte's avatar
Luc Yriarte committed
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
To create a `WellLog` record, below is a payload sample for the POST `/ddms/v3/welllogs` API. The response will contain an id you can use to create some bulk data.

```json
[
  {
    "acl": {
      "viewers": [
        "data.default.viewers@{{datapartitionid}}.{{domain}}"
      ],
      "owners": [
        "data.default.owners@{{datapartitionid}}.{{domain}}"
      ]
    },
    "data": {
      "Curves": [
        {
          "CurveID": "GR_ID",
          "Mnemonic": "GR",
          "CurveUnit": "{{datapartitionid}}:reference-data--UnitOfMeasure:m:",
          "LogCurveFamilyID": "{{datapartitionid}}:reference-data--LogCurveFamily:GammaRay:"
        },
        {
          "CurveID": "POR_ID",
          "Mnemonic": "NPOR",
          "CurveUnit": "{{datapartitionid}}:reference-data--UnitOfMeasure:m:",
          "LogCurveFamilyID": "{{datapartitionid}}:reference-data--LogCurveFamily:NeutronPorosity:"
        },
        {
          "CurveID": "Bulk Density",
          "Mnemonic": "RHOB",
          "CurveUnit": "{{datapartitionid}}:reference-data--UnitOfMeasure:m:",
          "LogCurveFamilyID": "{{datapartitionid}}:reference-data--LogCurveFamily:BulkDensity:"
        }
      ],
      "WellboreID": "{{datapartitionid}}:master-data--Wellbore:{{wellboreId}}:",
      "CreationDateTime": "2013-03-22T11:16:03Z",
      "VerticalMeasurement": {
        "VerticalMeasurement": 2680.5,
        "VerticalMeasurementPathID": "{{datapartitionid}}:reference-data--VerticalMeasurementPath:MD:",
        "VerticalMeasurementUnitOfMeasureID": "{{datapartitionid}}:reference-data--UnitOfMeasure:ft:"
      },
      "TopMeasuredDepth": 12345.6,
      "BottomMeasuredDepth": 13856.25,
      "Name": "{{welllogName}}",
      "ExtensionProperties": {
        "step": {
          "unitKey": "ft",
          "value": 0.1
        },
        "dateModified": "2013-03-22T11:16:03Z"
Luc Yriarte's avatar
Luc Yriarte committed
229
      }
Luc Yriarte's avatar
Luc Yriarte committed
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
    },
    "id": "{{datapartitionid}}:work-product-component--WellLog:{{welllogId}}",
    "kind": "osdu:wks:work-product-component--WellLog:1.0.0",
    "legal": {
      "legaltags": [
        "{{legaltags}}"
      ],
      "otherRelevantDataCountries": [
        "US",
        "FR"
      ]
    },
    "meta": [
      {
        "kind": "Unit",
        "name": "ft",
        "persistableReference": "{\"scaleOffset\":{\"scale\":0.3048,\"offset\":0.0},\"symbol\":\"ft\",\"baseMeasurement\":{\"ancestry\":\"Length\",\"type\":\"UM\"},\"type\":\"USO\"}",
        "propertyNames": [
          "stop.value",
          "elevationReference.elevationFromMsl.value",
          "start.value",
          "step.value",
          "reference.unitKey"
        ],
        "propertyValues": [
          "ft"
        ]
      },
Luc Yriarte's avatar
Luc Yriarte committed
258
      {
Luc Yriarte's avatar
Luc Yriarte committed
259
260
261
262
263
264
265
        "kind": "DateTime",
        "name": "datetime",
        "persistableReference": "{\"format\":\"yyyy-MM-ddTHH:mm:ssZ\",\"timeZone\":\"UTC\",\"type\":\"DTM\"}",
        "propertyNames": [
          "dateModified",
          "dateCreated"
        ]
Luc Yriarte's avatar
Luc Yriarte committed
266
      }
Luc Yriarte's avatar
Luc Yriarte committed
267
268
269
270
271
    ]
  }
]
```

272
273
274
275
276
277
278

### Run with Uvicorn

```bash
uvicorn app.wdms_app:wdms_app --port LOCAL_PORT
```

279
Then access app on `http://127.0.0.1:<LOCAL_PORT>/api/os-wellbore-ddms/docs`
280
281
282
283
284
285
286
287
288
289

### Run with Docker

#### Build Image

```bash
# Set IMAGE_TAG
IMAGE_TAG="os-wellbore-ddms:dev"

# Build Image
Luc Yriarte's avatar
Luc Yriarte committed
290
docker build -t=$IMAGE_TAG --rm . -f ./build/dockerfile --build-arg PIP_WHEEL_DIR=python-packages
291
292
293
294
295
296
```

#### Run Image

1. Run the image

Luc Yriarte's avatar
Luc Yriarte committed
297
    Replace the LOCAL_PORT value with a local port
298

Luc Yriarte's avatar
Luc Yriarte committed
299
300
    ```bash
    LOCAL_PORT=<local_port>
301

Luc Yriarte's avatar
Luc Yriarte committed
302
303
    docker run -d -p $LOCAL_PORT:8080 -e OS_WELLBORE_DDMS_DEV_MODE=1 -e USE_LOCALFS_BLOB_STORAGE_WITH_PATH=1 $IMAGE_TAG
    ```
304

305
2. Access app on `http://127.0.0.1:<LOCAL_PORT>/api/os-wellbore-ddms/docs`
306
307
308
309
310

3. The environment variable `OS_WELLBORE_DDMS_DEV_MODE=1` enables dev mode

4. Logs can be checked by running

Luc Yriarte's avatar
Luc Yriarte committed
311
312
313
    ```bash
    docker logs CONTAINER_ID
    ```
314
315
316
317
318
319
320
321
322
323
324
325

### Run Unit Tests Locally

```bash
# Install test dependencies
pip install -r requirements_dev.txt

python -m pytest --junit-xml=unit_tests_report.xml --cov=app --cov-report=html --cov-report=xml ./tests/unit
```

Coverage reports can be viewed after the command is run. The HMTL reports are saved in the htmlcov directory.

326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
### Run Integration Tests locally

This example runs basic tests using the local filesystem for blob storage and storage service. There's no search or entilements service, everything runs locally.  

First, create the temp storage folders and run the service.

```bash
mkdir -p tmpstorage
mkdir -p tmpblob
python main.py -e USE_INTERNAL_STORAGE_SERVICE_WITH_PATH $(pwd)/tmpstorage -e USE_LOCALFS_BLOB_STORAGE_WITH_PATH $(pwd)/tmpblob -e CLOUD_PROVIDER local
```

In another terminal, generate a minimum configuration file and run the integration tests.

```bash
cd tests/integration
python gen_postman_env.py --token $(pyjwt --key=secret encode email=nobody@example.com) --base_url "http://127.0.0.1:8080/api/os-wellbore-ddms" --cloud_provider "local" --data_partition "dummy"
pytest ./functional --environment="./generated/postman_environment.json" --filter-tag=basic
```

For more information see the [integration tests README](tests/integration/README.md)

348
349
### Port Forward from Kubernetes

Luc Yriarte's avatar
Luc Yriarte committed
350
351
 1. List the pods: `kubectl get pods`
 2. Port forward: `kubectl port-forward pods/POD_NAME LOCAL_PORT:8080`
352
 3. Access it on `http://127.0.0.1:<LOCAL_PORT>/api/os-wellbore-ddms/docs`
353

Luc Yriarte's avatar
Luc Yriarte committed
354
### Tracing
355

Luc Yriarte's avatar
Luc Yriarte committed
356
357
OpenCensus libraries are used to record incoming requests metrics (execution time, result code, etc...).
At the moment, 100% of the requests are saved.