Skip to content
GitLab
Commit eca7513d authored by Greg Harris's avatar Greg Harris
Browse files

Merge branch '46-system-maintenance-guide' into 'main' 

Resolve "Write System Maintenance guide"

Closes #46

See merge request !39
parents 4eb463d1 c4c6d0b7
Pipeline #98183 passed with stages
in 1 minute and 8 seconds
Hide whitespace changes
Inline Side-by-side
README.md
View file @ eca7513d
# LAS Loader utility for Wellbore DDMS
# Data Loader Utility for Wellbore DDMS
## :exclamation: This is currently in development
## Description
1. This will be the OSDU LAS loader utility to support Wellbore DDMS.
1. This will be the OSDU Data loader utility to support Wellbore DDMS.
2. This is a wrapper behind the Wellbore DDMS [API](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/blob/master/spec/generated/openapi.json)
## Install from Package Registry
### Before installation:
- Please ensure your Python version is 3.8 or newer and that Python has been added to the PATH (in your system's environment variables).
- Ensure Python 3.8 or newer is installed and that Python has been added to the PATH (in your system's environment variables).
- Confirm you have `pip` installed by running the following command (it should be installed automatically when you installed Python) - `pip --version`
- If installing `wbdutil` within a virtual environment, first install `pipenv` using the following command - `pip install pipenv`
- Note: There is a bug in `pipenv` version 2021.11.23 which will cause the `wbdutil` installation command to fail. Ensure your version number is different to this (e.g. `pip install pipenv=="2021.11.15"`).
- Note: There is a bug in `pipenv` version 2021.11.23 which will cause the `wbdutil` installation command to fail. Ensure your version number is different to this (e.g. `pip install pipenv=="2021.11.15"` or alternatively a newer version).
### Installation:
The simplest way to install `wbdutil` is from the community package registry.
......@@ -28,6 +29,7 @@ pip install wbdutil --extra-index-url https://community.opengroup.org/api/v4/pro
pipenv install wbdutil --extra-index-url https://community.opengroup.org/api/v4/projects/801/packages/pypi/simple --skip-lock
```
## Usage
The `wbdutil` package has a command line interface of the same name `wbdutil` which has the general syntax:
......@@ -45,10 +47,10 @@ There are several groups:
* `search`: Search for a wellbore given a well name.
* `update`: Update the existing bulk data for a given welllog
The `wbdutil` reads configuration information from a [configuration file](#config-file).
The `wbdutil` requires configuration information, read from a [configuration file](#config-file).
The path to this file must either be provided in the environment variable `CONFIGPATH` or as a command line option.
All the commands that connect to an OSDU instance require a bearer token.
All the commands that connect to an OSDU instance (e.g. commands within the `ingest` group) require a bearer token.
This must be provided either in the environment variable `OSDUTOKEN` or as a command line option.
### Environment variables
......@@ -56,12 +58,12 @@ This must be provided either in the environment variable `OSDUTOKEN` or as a com
| Varaible name | overriding command option | Comment |
| ------------- | --------------------------| ------- |
| OSDUTOKEN | `--token` `-t` | The JWT required to authenticate against an OSDU instance. |
| CONFIGPATH | `--config_path` `-c` | The path to the configuration file |
| CONFIGPATH | `--config_path` `-c` | The path to the configuration file. |
### Config file
The wbdutil requires a configuration file that has the following JSON structure:
The `wbdutil` requires a configuration file that has the following JSON structure:
```
{
"base_url": "https://osdu-ship.msft-osdu-test.org",
......@@ -206,74 +208,3 @@ There are a limited number of mapping functions available these are listed below
| `las2osdu_curve_uom_converter(unit, data_partition_id)`| `welllog_mapping` | This function converts a LAS format unit of measure to an OSDU format UoM. |
These are hard coded functions, so a change request will need to be raised if additional functions are required. We have avoided user defined functions, because such functions represent a small security risk.
## Development
The following instructions are provided as guidance for setting up a development environment for wbdutil. For development work we use a virtual environment managed by pipenv. To execute the application and run the tests you will need to install Python 3.8+ and pipenv.
The source code and unit test are stored in the `src` folder.
To download the package dependencies for the virtual environment change to the src folder and run:
```
pipenv install --dev
```
To run the tests and flake8 linting:
```
pipenv run flake8
pipenv run pytest
```
Alternatively, an interactive shell in the virtual environment can be created by running:
```
pipenv shell
```
### Adding package dependencies
Package dependencies are added using the pipenv command. Dependencies required for only the unit tests should be added to the `[dev-packages]` of the `Pipfile` by using:
```
pipenv install --dev package_name
```
Production packages should be installed using:
```
pipenv install package_name
```
## Deploy
### Install from repository source
The wbdutil module can be installed in edit mode on the command line by running the following command in the root folder of the repository:
```
pip install -e .
```
The wbdutil CLI can then be accessed using the command:
```
wbdutil
```
Or alternatively using a virtual environment, install with:
```
pipenv install -e .
```
This installs the package in a virtual environment. The CLI can then be run using:
```
pipenv run wbdutil
```
### Publish to the Package Registry
Packages are published to the community package registry via the CI pipeline. Package publish is a manually triggered job. To run it navigate to the CI/CD jobs page of the `wbdutil` module:
[https://community.opengroup.org/osdu/platform/data-flow/data-loading/wellbore-ddms-las-loader/-/jobs](https://community.opengroup.org/osdu/platform/data-flow/data-loading/wellbore-ddms-las-loader/-/jobs).
Run the publish job for the build you want to publish. It will build and upload `wbdutil` with a specific package version number. The package version number is the `__VERSION__` found in the `__init__.py` file for the module followed by a build tag. The build tag is based upon the CI build_id:
1. The default build tag is an alpha release of the form 1.2.3a456
1. If the repo branch has been tagged with a string starting with `BETA` then the beta release form is used 1.2.3b456.
1. If the repo branch has been tagged with a string starting with `RELEASE` then the primary release form is used 1.2.3.456.
csp-auth-postman/Azure-Auth.postman_collection.json
View file @ eca7513d
......@@ -51,7 +51,7 @@
},
{
"key": "refresh_token",
"value": "0.ATcA01-XWHdJ0ES-qDevC6rBANMOMqvdnJhHjjwqZXgAGDs3AMU.AgABAAAAAAD--DLA3VO7QrddgJg7WevrAgDs_wQA9P8Iv1xNr9laoueeoxis1kIpRGOOEOgp0NprM9u_OUTGQPWlImcBbNRKJOH367iIzEYbRAACtXv_9P9gh5vYasAk5kn6GT0wnHlfHyEn3sCINU8JblkD_At76rnr4Q46yHLj1LXRbum4ILgLm6L8t-xjkIWmaBCsO3oYJfHkosPMroAGDheoo4BHS25HbU_HCqUpIMWP7e_JMyVXXYnUCjNtGIoLdwrtLfvW7d4i4B15FZdjv3ACcV-hrpzxGzdv5F8F0VQuNpdVM1BMBvU09npoS5eC8YLyRvzEiJuO1JJZKfCWzmnJT7EF_V_xEgwavHDO6Ux_ATbL-6dFL_QuCOpJ_lIjmGiS1JR-9PvLU1fjt-aN86SKUzN6Jn4wKnZfuuz1imXJ24QXYR0Fhc7D7DRXfYe0O4N45m35ICmhD2sh5ngro_xtQ329SEEEWDjMdk8g3Naop0WLlpnAPrtlND4mAB6l_GFGZzKR4_ThwuGHN6dLvGjNvDVsvEWr3ltLdUaWFcdUfXUwBUzZRqyTW9YfQMrw25uUB8_NCsaJEfoxOaCZX9iASQX_lIZL-DuMrNgCfuDACajZOibcJPkmzhOKtpCxl-zWzEHYyQY3WM64zw8DCBhPwNee-q98qvBFxqUcS-FMZPcYBSyt3ElJ2cnnto9DT-KN6CxB_r3fx3MzrVkJB3Pm3Zw26o99v0Iu4mp47HkEFH__uimjn7pjbK2sOxZ5UpOjXm88h8UvtCknViRIQPhPR1V94_Uk755BA4JDjwyd4aDO5YVCTSYkst1S5EurxlBa6iWphIpLJLPe-VqjZYPH4v0iC1MVXDRlcUlCTKvpV1Ew7mAx4aKi9MC4E6kc8zqAQHXYzgOKPPLVirCK8Xtl3jlxxLPCb185F6PFpdGUXdi9kbmeXXfl0IRhUS-_SNDV_wYaf-4bx1TkguFkPHLtOx6aKPDtpg31pMz2o_jFnWvEqPlETseI27iPDohj_Ygx7Z-iYy_XRQ-uVsQBmBLG6chbyzvS5bK1VxkzK40v08UribbYkfmJdO2NE25bXWU",
"value": "0.ATcA01-XWHdJ0ES-qDevC6rBANMOMqvdnJhHjjwqZXgAGDs3AMU.AgABAAAAAAD--DLA3VO7QrddgJg7WevrAgDs_wQA9P_eICN-tIk1uly__kJcuTiotmHD2NSmtT3GPQvWD_yQB6-9fowg1bMAVDOjLn2nUwZyHkgXVCCR6SSKOxpunlpvaL588EMZzARSjVF63cprhgl-K6YyuK3ZQcPRRIBNSURxYgkK_XJTZcw_Ozx8xZDBhadmIhD7QP9El08u13ntLBW_Yuz2s-zBaqCCQbmtlKVgP8_ukJbrAEP5r7Q_5YDRm7ygUM-2UeNaQ0V30gv4HKuo0uwJbW61lGajFsUewz9lIA927j8nBiCPqB-bPtUSfb3FsgOpxql50Ppvans09Dj5EoB9wdYcmgZkC9Q6PlNzvUpU1HpYidsqOKQ5V0KIW2volTryoq42ZtETb0bfj7gkkPaUtpYfQH22M_HUFgc2OS5ciO1oAQleeiwxobBbTGgI_Cn9_kA8_BaIy0F1sn1-qX9Qaq6FgYzSdvIUiYX-5wVsmXCAndXW4Y0cbuGnA2tDb6BxtiWwA7x4hb5ja6RTZ69svtaY9eQ2OMEx5mu0ccymzlC3KKaRkuu8PbUl3X2cuhIraFa08SlaSQEjuoqImCFABySPSFZNeFbTyadAlzHYJH0DqUxi8cyW6Ofi5mI7UJz6GTShG9M4Iw6X0eEMOYrGgEAg1NbakSDb24Si4cjittm5D1vGCIdoGHIHaeWC2jWIiv6VZqizggSZOOARN_Ga-oj3anHeSmULNwdGl6texV_XC8ZQrGtRK3GuxP1CF-_OJ2EBoQri5BTyscMsQGZ9DidjbqobhuKlceP-IcKQcGmWmh7_2d4ez0xuQ5FTeswv4-uxQn-U93XBtDyHvM7SzIGir3jUmR2SDI_NrWiSomEHbQVqA1eTps8go3bgwoZOMFuo4GyE0CKrLw27Wn0OkFMsjh5z0ECAwty4c8SrSvOA3IeKihEnFecbdaUtq7JPO0SjeSAk67IV5R_MqDwaj-v9HfdxdBo5-gV8",
"type": "text"
}
]
......
doc/SMG.md 0 → 100644
View file @ eca7513d
- [System Maintenance Guide - Wellbore DDMS Data Loader](#system-maintenance-guide---wellbore-ddms-data-loader)
- [Overview](#overview)
- [Development Access and Contacts](#development-access-and-contacts)
- [Key Contacts](#key-contacts)
- [Development Setup](#development-setup)
- [Package Dependencies](#package-dependencies)
- [OSDU Authorization Tokens](#osdu-authorization-tokens)
- [Refresh Tokens](#refresh-tokens)
- [Linting and Tests](#linting-and-tests)
- [Architecture](#architecture)
- [Repository Structure](#repository-structure)
- [src](#src)
- [wbdutil](#wbdutil)
- [Branching and Merge Policies](#branching-and-merge-policies)
- [Branching](#branching)
- [Merge Requests](#merge-requests)
- [Package Deployment and Installation](#package-deployment-and-installation)
- [setup.py](#setuppy)
- [Adding New Dependencies](#adding-new-dependencies)
- [Install from Repository Source](#install-from-repository-source)
- [Publish to the Package Registry](#publish-to-the-package-registry)
- [Automated Steps](#automated-steps)
- [Manual Step](#manual-step)
# System Maintenance Guide - Wellbore DDMS Data Loader
## Overview
This system is a wrapper behind the [Wellbore DDMS API](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/blob/master/spec/generated/openapi.json). It exists as a utility to support the loading of data from files (e.g. LAS) into an OSDU instance.
The system is implemented as a command line application that can be installed by following the instructions in the repository's [README.md](../README.md) file.
## Development Access and Contacts
The system can be managed completely via the [GitLab project](https://community.opengroup.org/osdu/platform/data-flow/data-loading/wellbore-ddms-data-loader). Developer access to this project can be requested from one of the contacts below.
### Key Contacts
- **Chad Leong (cleong4@wellbarrier.slb.com)** is the product owner for this system. He can grant required access for development activities (i.e. developer access to the repository).
- The **R3 Data Loading** team are the current users of the system. If Chad Leong is unavailable someone in the *#2_2_2_r3_data_loading* channel (ID: CTZK7DWP2) on *The OSDU Forum* Slack workspace should be able to help.
## Development Setup
The following instructions are provided as guidance for setting up a development environment for *wbdutil*. For development work we use a virtual environment managed by *pipenv*. To execute the application and run the tests you will need to install **Python 3.8+** and **pipenv**.
The source code and unit tests are stored in the `src` folder.
### Package Dependencies
To download and install the package dependencies for the virtual environment change the current working directory to the src folder and run:
```
pipenv install --dev
```
Package dependencies are added using the *pipenv* command. Dependencies required for only the unit tests should be added to the `[dev-packages]` of the `Pipfile` by using:
```
pipenv install --dev <package_name>
```
Production packages should be installed using:
```
pipenv install <package_name>
```
### OSDU Authorization Tokens
The Wellbore DDMS API that this system communicates with uses token based authentication.
A token can be acquired using the POST request defined in the 'Azure Auth' Postman collection (see file `csp-auth-postman/Azure-Auth.postman_collection.json`). This JSON file can be imported into a Postman workspace as a collection and the 'Get Bearer Token - Wellbore DDMS' request run from there.
#### Refresh Tokens
The body of the 'Get Bearer Token' request requires a refresh token to continue working - these expire after a period of ~90 days so will need to be updated.
There are two options for getting this:
- Each time the POST request for a new bearer token is made, a new refresh token will be returned as well. This refresh token can be copied into the body of the request.
- If the refresh token has expired, meaning the POST request for a new token cannot be made, the Cloud Service Provider (CSP) will need to be contacted and they can provide a new token. This could be Microsoft, IBM, Google Cloud Platform (GCP) or Amazon Web Services (AWS).
### Linting and Tests
'flake8' linting is configured within the [src/.flake8](../src/.flake8) file.<br />
'pytest' finds all tests in the `src/test` folder and runs them.
To run the flake8 linting and tests directly:
```
pipenv run flake8
pipenv run pytest
```
Alternatively, you may activate the virtual environment and run the commands within that:
```
pipenv shell
flake8
pytest
```
## Architecture
The system can be understood to be formed of a series of "layers":
- **Command Interface:** Top level code related to the CLI (e.g. `wbdutil.py` file).
- **Command Processors:** Action the commands called from the CLI and call the service layer.
- **Service Layer:** Main business logic, including data mapping.
- **OSDU and OS Wrappers:** Connect to and call OSDU APIs as well as file reading.
- **Common:** Code used throughout all other layers (e.g. config definition, custom exceptions, utilities).
- **Test Code:** Unit tests for the code.
![Wellbore DDMS Data Loader Architecture](images/architecture-data-loader.png)
## Repository Structure
### src
The `src` directory contains all the code and tests for this system.
Within the `src` folder:
- `wbdutil` directory - source code for the application
- `test` directory - unit tests for all modules within `wbdutil`
- `.flake8` file - flake8 linting configuration file
- `Pipfile` file - pipfile for managing virtual environment
#### wbdutil
The `wbdutil` directory contains the entry point for the command line application (`wbdutil.py`) and a range of supporting modules as defined in the [Architecture](#architecture) section above. It is structured as follows:
```
wbdutil/
|--- wdbutil.py
|--- commands/
|--- common/
|--- service/
|--- wrapper/
```
All supporting modules within the directories defined above are documented using docstrings.
## Branching and Merge Policies
### Branching
For each issue being worked on, a new branch based on `main` should be created for all work related to that task to be committed to.
Branch naming convention: "\<issue number>-summary-of-task" (e.g. *46-write-system-maintenance-guide*)
### Merge Requests
When the work is complete a merge request (MR) can be opened to merge the development branch into `main`. Once the code has been reviewed and the MR approved by at least one other developer, the MR can be completed.
The following guidance should be followed where possible:
- The creator of a MR (i.e. the developer that completed the work) should not approve their own MR (unless the work completed is very minor).
- The creator of a MR should be the one to complete the MR once approved (not the person that has reviewed/approved it).
## Package Deployment and Installation
### setup.py
The `setup.py` file at the root of this repository defines how the module is packaged. It defines the package version (which is generated automatically as defined in the [Manual Step](#manual-step) subsection below) and package dependencies.
The majority of this file should not require editing but the `install_requires` section of the main `setup` function will need updating when new package dependencies are introduced.
#### Adding New Dependencies
When new dependencies are added to the Pipfile, they will also need to be added to the `setup.py` file so that they are installed as part of the package installation. This can be done manually or automatically.
- To add dependencies manually, add a new element of the form `"package_name==<version>"` within the `install_requires` list in the `setup()` function.
- To add dependencies automatically, first ensure the `setup.py` file is in the same location as the `Pipfile.lock` file. Navigate to this directory and run the command `pipenv-setup sync`. Use this command with care and always double check any changes that have been made to `setup.py` before committing them.
More information can be found in the Python documentation [here](https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#install-requires).
### Install from Repository Source
The *wbdutil* module can be installed in edit mode on the command line by running the following command in the root folder of the repository:
```
pip install -e .
```
The *wbdutil CLI* can then be accessed using the command:
```
wbdutil
```
Or alternatively using a virtual environment, install with:
```
pipenv install -e .
```
This installs the package in a virtual environment. The CLI can then be run using:
```
pipenv run wbdutil
```
### Publish to the Package Registry
Packages are published to the community package registry via the continuous integration (CI) pipeline. Full details of the pipeline can be found in the [.gitlab-ci.yml](../.gitlab-ci.yml) file.
#### Automated Steps
On completion of any merge request (i.e. when a development branch is merged into `main` branch) the following steps are run automatically:
1. Linting - configured within the [src/.flake8](../src/.flake8) file.
2. Tests - 'pytest' finds all tests within the `src/test` folder and runs them.
3. Packaging/Building - using the [setup.py](../setup.py) file, the application is packaged into a wheel (.whl) file.
#### Manual Step
Package publish is a manually triggered job.
To run it, navigate to the 'CI/CD Jobs' page of the Wellbore DDMS Data Loader GitLab project:
[https://community.opengroup.org/osdu/platform/data-flow/data-loading/wellbore-ddms-las-loader/-/jobs](https://community.opengroup.org/osdu/platform/data-flow/data-loading/wellbore-ddms-las-loader/-/jobs).
Run the publish job for the build you want to publish. It will build and upload `wbdutil` with a specific package version number. The package version number is the `__VERSION__` found in the `__init__.py` file for the module followed by a build tag. The build tag is based upon the CI `build_id`:
1. The default build tag is an alpha release of the form 1.2.3a456
1. If the repo branch has been tagged with a string starting with `BETA` then the beta release form is used 1.2.3b456.
1. If the repo branch has been tagged with a string starting with `RELEASE` then the primary release form is used 1.2.3.456.
doc/images/architecture-data-loader.drawio 0 → 100644
View file @ eca7513d
<mxfile host="Electron" modified="2022-03-09T11:19:05.384Z" agent="5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/14.1.8 Chrome/87.0.4280.88 Electron/11.1.1 Safari/537.36" etag="CAdpSeD2na2DaKL-Zz5w" version="14.1.8" type="device"><diagram id="N4hjGRzpWYw_M2NuAcfk" name="Page-1">7ZrbcpswEIafxpfpAAJsX6ZOephJO5lxMmmuOopRgFawVMg27tNXsoWxLDuhjTHU4yuzqwPS/voWJNxDo6T4yHAWfYGA0J5jBUUPXfUcx0aOL36kZ7HyDCzlCFkcqEqVYxz/JsppKe80DkiuVeQAlMeZ7pxAmpIJ13yYMZjr1Z6B6nfNcEgMx3iCqel9iAMeqVl4VuX/ROIwKu9sW6okwWVl5cgjHMB8w4Wue2jEAPjqKilGhMrglXFZtfuwp3Q9MEZSXqdBcR8hsIdXd/jidk6+D/zH8dcL11WD44tyxiQQAVAmMB5BCCmm15X3PYNpGhDZrSWsqs4NQCactnD+IJwvlJp4ykG4Ip5QVUqKmH+Tzd95ynpUncnrq2LTWJRGytlio5E0H8v+pFE1W1plu9X85KT2xk25cpiyCXkpWI5agJiFhL9QEflrfQUYBBIiRiQaMkIxj2f6SLBaoeG6XiWiuFA6/o2mapgzTKfqVobIuoTzKOZknOHl7OcCZF2u55jSEVBgwk4hlUtgvZirEM8I46R4OchmSFQDNFCULMrEoex5BZ3nK1+0AZxTknjwMNo7ouhTLuMBYlKb4fR/TaEsuMiXq/5SVLDdrKgKxVUof0eQJJCWfYmhrbpbFb5NqAMI4Vu6EMgyhbCdHUL4TemAvO6kKPufUpR1vBSFamaoPYvgOAkKNUTWgwAiIyzvKluDLbZsr3W2bCMoR2TL3iCr4uw1tjSyKtCO8Piv+/Rvla0OvdB1PlvWVbTVbOk2lC3HhM1iER7HusELwrqaNLdfSNpPms4ZsfqIef8DYsjcIJ0fg29V1G1TUa+hpCk3bzgNRPktgwnJc+ju66bbvcx5ft2sz5lfkzOvTc78xjn7nHLCniUmHcXM6R5mlhGUM2b7gtWviVmrR7r9t2M22IVZ2ctT6bjPtZ3A03bF1+ATGHF9WeScwU+ydXy840QZ0zhMhTkhEnjhkFDGE0wvVUESB8Fype5CWl+9B3l46lD7JtS7jqPdxk6jB8YSEG8gs+XHMdE0zab8dMWwa6TY4a6PA42pMTTUGGFK89OVADmvSzA4pgSOmRNF8jphBbzOKWBCcOIKDHUFnH69h0JjCiDzS8ppK9DvGgOoqQ3QHcll6xEEnd34bJ/MOm3ve1zz1EeG8YR58G1dgqFnSOAdBgdhVn9VWpZt/OELXf8B</diagram></mxfile>
\ No newline at end of file
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment