Commit 55013bc9 authored by Niall McDaid's avatar Niall McDaid Committed by Niall McDaid
Browse files

Move dev setup info to SMG

parent 153a45b8
......@@ -66,6 +66,7 @@ The wbdutil requires a configuration file that has the following JSON structure:
{
"base_url": "https://osdu-ship.msft-osdu-test.org",
"data_partition_id": "opendes",
"acl_domain": "contoso.com",
"legal":
{
"legaltags": ["opendes-public-usa-dataset-7643990"],
......@@ -77,203 +78,8 @@ The wbdutil requires a configuration file that has the following JSON structure:
"viewers": [ "data.default.viewers@opendes.contoso.com" ],
"owners": [ "data.default.owners@opendes.contoso.com" ]
}
},
"wellbore_mapping": {},
"welllog_mapping": {}
}
```
The `base_url` and `data_partition_id` must be correct for the OSDU instance that you want to connect to.
`wellbore_mapping` and `welllog_mapping` are optional features described in the [custom mappings](#custom-mappings) section.
#### Custom mappings
**Custom mappings are an advanced feature of `wbdutil` that require knowledge of both `lasio.LASFile` and OSDU data object schemas and should be used with care.**
The configuration file can be used to define optional custom mappings from `lasio.LASFile` data objects
to OSDU wellbore and welllog objects of a specified kind.
It is recommended that a new mapping is thoroughly tested using the `parse` command group, before upload to OSDU.
There are 2 mapping types `wellbore_mapping` and `welllog_mapping` both must contain a `kind` and a `mapping` attribute.
If `wellbore_mapping` or `welllog_mapping` are not defined in the configuration file `wbdutil` will use the default mappings for the well bore and/or well log.
The `kind` attribute defines the target OSDU data type (kind), for example `osdu:wks:work-product-component--WellLog:1.1.0`.
The `mapping` type describes how data in the incoming `lasio.LASFile` object should be transformed into the target OSDU data kind.
Here is an example mapping for a welllog that could be added to a configuration file.
```
{
"welllog_mapping": {
"kind": "osdu:wks:work-product-component--WellLog:1.1.0",
"mapping":
{
"acl.viewers": "CONFIGURATION.data.default.viewers",
"acl.owners": "CONFIGURATION.data.default.owners",
"legal.legaltags": "CONFIGURATION.legal.legaltags",
"legal.otherRelevantDataCountries": "CONFIGURATION.legal.otherRelevantDataCountries",
"legal.status": "CONFIGURATION.legal.status",
"data.ReferenceCurveID": "curves[0].mnemonic",
"data.WellboreID": {
"type": "function",
"function": "get_wellbore_id",
"args": []
},
"data.Curves": {
"type": "array",
"source": "curves",
"mapping": {
"CurveID": "mnemonic",
"Mnemonic": "mnemonic",
"CurveUnit": {
"type": "function",
"function": "las2osdu_curve_uom_converter",
"args": [
"unit",
"CONFIGURATION.data_partition_id"
]
}
}
}
}
}
}
```
The simple data mappings take the form of a key and string value pair.
The key (string to the left of the semi-colon) is the target field within the OSDU data kind and
the value string defines the source of the data. For example:
`"data.ReferenceCurveID": "curves[0].mnemonic"` will set the `data.ReferenceCurveID` field of the output OSDU object
to the value of the `mnemonic` field of the first element in the `curves` array of the input `lasio.LASFile` object.
The `CONFIGURATION` keyword indicates that data should be taken from the configuration file, for example:
`"acl.viewers": "CONFIGURATION.data.default.viewers"` will set the `acl.viewers` field of the output OSDU object
to the value of the `data.default.viewers` field of the configuration. The simple mapping form supports the direct copying of
all objects including arrays from the incoming LAS data to the output OSDU data kind.
There are often more complex transformations that need to be performed on the incoming data,
`wbdutil` supports two types of complex mapping `array` and `function`.
The `function` complex mapping type makes a call to a hard coded function to perform a transformation on the incoming data.
For example:
```
"CurveUnit": {
"type": "function",
"function": "las2osdu_curve_uom_converter",
"args": [
"unit",
"CONFIGURATION.data_partition_id"
]
}
```
This will set the value of the `CurveUnit` field to the output of the function `las2osdu_curve_uom_converter` using the input arguments in the args array (in this case `unit` from the input array element and `data_partition_id` from the configuration file).
The second complex mapping is `array` this should be used if the elements of an incoming array need to be changed in some way.
This could be a field name change, a change in the object structure or to call a function on specific data within each element.
Here is an example:
```
{
"data.Curves": {
"type": "array",
"source": "curves",
"mapping": {
"CurveID": "mnemonic",
"Mnemonic": "mnemonic",
"CurveUnit": {
"type": "function",
"function": "las2osdu_curve_uom_converter",
"args": [
"unit",
"CONFIGURATION.data_partition_id"
]
}
}
}
}
```
This mapping will iterate over the `curves` array of the input `lasio.LASFile` object and apply an
inner mapping to each element in the array.
In this case the inner mapping is defined so that the `mnemonic` field of the `curve` element is
mapped to both the `CurveID` and `Mnemonic` output fields, and the `CurveUnit` output field is set
to the return value of the function `las2osdu_curve_uom_converter`
that takes the `unit` field of array element and the `data_partition_id` (from configuration) as arguments.
The resulting output array is mapped to the `data.Curves` field of the output OSDU kind.
An example configuration file that is setup for the preship OSDU instance is given in `src/example_opendes_configuration.json`,
it also contains example custom mappings for the `osdu:wks:master-data--Wellbore:1.0.0` wellbore kind and the `osdu:wks:work-product-component--WellLog:1.1.0` welllog kind.
There are a limited number of mapping functions available these are listed below:
| Function name | Mapping type | Purpose |
| ------------------------------------------------------ | -------------------| ----------------------------------------------------- |
| `build_wellbore_name_aliases(uwi, data_partition_id)` | `wellbore_mapping` | Constructs a name alias object from the LAS UWI and the data partition id. |
| `get_wellbore_id()` | `welllog_mapping` | Returns the wellbore id from the wellbore that corresponds to the welllog |
| `las2osdu_curve_uom_converter(unit, data_partition_id)`| `welllog_mapping` | This function converts a LAS format unit of measure to an OSDU format UoM. |
These are hard coded functions, so a change request will need to be raised if additional functions are required. We have avoided user defined functions, because such functions represent a small security risk.
## Development
The following instructions are provided as guidance for setting up a development environment for wbdutil. For development work we use a virtual environment managed by pipenv. To execute the application and run the tests you will need to install Python 3.8+ and pipenv.
The source code and unit test are stored in the `src` folder.
To download the package dependencies for the virtual environment change to the src folder and run:
```
pipenv install --dev
```
To run the tests and flake8 linting:
```
pipenv run flake8
pipenv run pytest
```
Alternatively, an interactive shell in the virtual environment can be created by running:
```
pipenv shell
```
### Adding package dependencies
Package dependencies are added using the pipenv command. Dependencies required for only the unit tests should be added to the `[dev-packages]` of the `Pipfile` by using:
```
pipenv install --dev package_name
```
Production packages should be installed using:
```
pipenv install package_name
```
## Deploy
### Install from repository source
The wbdutil module can be installed in edit mode on the command line by running the following command in the root folder of the repository:
```
pip install -e .
```
The wbdutil CLI can then be accessed using the command:
```
wbdutil
```
Or alternatively using a virtual environment, install with:
```
pipenv install -e .
```
This installs the package in a virtual environment. The CLI can then be run using:
```
pipenv run wbdutil
```
### Publish to the Package Registry
Packages are published to the community package registry via the CI pipeline. Package publish is a manually triggered job. To run it navigate to the CI/CD jobs page of the `wbdutil` module:
[https://community.opengroup.org/osdu/platform/data-flow/data-loading/wellbore-ddms-las-loader/-/jobs](https://community.opengroup.org/osdu/platform/data-flow/data-loading/wellbore-ddms-las-loader/-/jobs).
Run the publish job for the build you want to publish. It will build and upload `wbdutil` with a specific package version number. The package version number is the `__VERSION__` found in the `__init__.py` file for the module followed by a build tag. The build tag is based upon the CI build_id:
1. The default build tag is an alpha release of the form 1.2.3a456
1. If the repo branch has been tagged with a string starting with `BETA` then the beta release form is used 1.2.3b456.
1. If the repo branch has been tagged with a string starting with `RELEASE` then the primary release form is used 1.2.3.456.
The `base_url`, `data_partition_id` and `acl_domain` must be correct for the OSDU instance you want to connect to. An example configuration file that is setup for the preship OSDU instance is given in `src/example_opendes_configuration.json`.
- [System Maintenance Guide - Wellbore DDMS Data Loader](#system-maintenance-guide---wellbore-ddms-data-loader)
- [Overview](#overview)
- [Architecture](#architecture)
- [Development Access and Contacts](#development-access-and-contacts)
- [Key Contacts](#key-contacts)
- [Release and Deployment Responsibilities](#release-and-deployment-responsibilities)
- [CI and CD](#ci-and-cd)
- [Software Dependencies](#software-dependencies)
- [Packages](#packages)
# System Maintenance Guide - Wellbore DDMS Data Loader
## Overview
This system is a wrapper behind the [Wellbore DDMS API](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/blob/master/spec/generated/openapi.json). It exists as a utility to support the loading of data from files (e.g. LAS) into an OSDU instance.
The system is implemented as a command line application that can be installed by following the instructions in the repository's [README.md](README.md) file.
## Architecture
## Development Access and Contacts
The system can be managed completely via the [GitLab project](https://community.opengroup.org/osdu/platform/data-flow/data-loading/wellbore-ddms-data-loader). Developer access to this project can be requested from one of the contacts below.
### Key Contacts
- **Chad Leong (cleong4@wellbarrier.slb.com)** is the product owner for this system. He can grant required access for development activities (i.e. developer access to the repository).
- The **R3 Data Loading** team are the current users of the system. If Chad Leong is unavailable someone in the *#2_2_2_r3_data_loading* channel (ID: CTZK7DWP2) on *The OSDU Forum* Slack workspace should be able to help.
## Release and Deployment Responsibilities
Continuous integration (CI) and continuous deployment (CD) pipelines are set up in the project on GitLab. These are set up to run automatically after every merge request is completed (i.e. when a development branch is merged into `main` branch).
It is therefore the responsibility of the developer and the approver of a merge request to ensure these pipelines are running and the system is being successfully deployed.
### CI and CD
On completion of any merge request the following steps are run:
1. Linting - 'flake8' linting is configured within the [src/.flake8](src/.flake8) file.
2. Tests - 'pytest' finds all tests within the `src/test` folder and runs them.
3. Packaging - using the [setup.py](setup.py) file, the application is packaged into a wheel (.whl) file.
4. Deployment - the new packaged application is deployed to the Open Group's PyPI registry.
Full details of the pipeline can be found in the [.gitlab-ci.yml](.gitlab-ci.yml) file.
## Software Dependencies
Developers and users should install the following:
- Python (version >=3.8)
### Packages
Developers: A Pipfile is used for managing package dependencies. The current Pipfile containing the most up-to-date list of packages used can be found [here](src/Pipfile). All required packages can be installed and maintained as described in the *Development* section of the repository's [README.md](README.md) file.
Users: All package dependencies will be installed automatically when the command line app is installed from the package registry.
- [System Maintenance Guide - Wellbore DDMS Data Loader](#system-maintenance-guide---wellbore-ddms-data-loader)
- [Overview](#overview)
- [Development Access and Contacts](#development-access-and-contacts)
- [Key Contacts](#key-contacts)
- [Development Setup](#development-setup)
- [Package Dependencies](#package-dependencies)
- [OSDU Authorization Tokens](#osdu-authorization-tokens)
- [Linting and Tests](#linting-and-tests)
- [Architecture](#architecture)
- [Repository Structure](#repository-structure)
- [Branching and Merge Policies](#branching-and-merge-policies)
- [Package Deployment and Installation](#package-deployment-and-installation)
- [Install from Repository Source](#install-from-repository-source)
- [Publish to the Package Registry](#publish-to-the-package-registry)
- [Automated Steps](#automated-steps)
- [Manual Step](#manual-step)
# System Maintenance Guide - Wellbore DDMS Data Loader
## Overview
This system is a wrapper behind the [Wellbore DDMS API](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/blob/master/spec/generated/openapi.json). It exists as a utility to support the loading of data from files (e.g. LAS) into an OSDU instance.
The system is implemented as a command line application that can be installed by following the instructions in the repository's [README.md](../README.md) file.
## Development Access and Contacts
The system can be managed completely via the [GitLab project](https://community.opengroup.org/osdu/platform/data-flow/data-loading/wellbore-ddms-data-loader). Developer access to this project can be requested from one of the contacts below.
### Key Contacts
- **Chad Leong (cleong4@wellbarrier.slb.com)** is the product owner for this system. He can grant required access for development activities (i.e. developer access to the repository).
- The **R3 Data Loading** team are the current users of the system. If Chad Leong is unavailable someone in the *#2_2_2_r3_data_loading* channel (ID: CTZK7DWP2) on *The OSDU Forum* Slack workspace should be able to help.
## Development Setup
The following instructions are provided as guidance for setting up a development environment for *wbdutil*. For development work we use a virtual environment managed by *pipenv*. To execute the application and run the tests you will need to install **Python 3.8+** and **pipenv**.
The source code and unit tests are stored in the `src` folder.
### Package Dependencies
To download and install the package dependencies for the virtual environment change the current working directory to the src folder and run:
```
pipenv install --dev
```
Package dependencies are added using the *pipenv* command. Dependencies required for only the unit tests should be added to the `[dev-packages]` of the `Pipfile` by using:
```
pipenv install --dev <package_name>
```
Production packages should be installed using:
```
pipenv install <package_name>
```
### OSDU Authorization Tokens
The Wellbore DDMS API that this system communicates with uses token based authentication
### Linting and Tests
'flake8' linting is configured within the [src/.flake8](../src/.flake8) file.<br />
'pytest' finds all tests in the `src/test` folder and runs them.
To run the flake8 linting and tests directly:
```
pipenv run flake8
pipenv run pytest
```
Alternatively, you may activate the virtual environment and run the commands within that:
```
pipenv shell
flake8
pytest
```
## Architecture
Can be a brief section.
Include a listing of the purposes of modules in here.
## Repository Structure
## Branching and Merge Policies
## Package Deployment and Installation
### Install from Repository Source
The *wbdutil* module can be installed in edit mode on the command line by running the following command in the root folder of the repository:
```
pip install -e .
```
The *wbdutil CLI* can then be accessed using the command:
```
wbdutil
```
Or alternatively using a virtual environment, install with:
```
pipenv install -e .
```
This installs the package in a virtual environment. The CLI can then be run using:
```
pipenv run wbdutil
```
### Publish to the Package Registry
Packages are published to the community package registry via the continuous integration (CI) pipeline. Full details of the pipeline can be found in the [.gitlab-ci.yml](../.gitlab-ci.yml) file.
#### Automated Steps
On completion of any merge request (i.e. when a development branch is merged into `main` branch) the following steps are run automatically:
1. Linting - configured within the [src/.flake8](../src/.flake8) file.
2. Tests - 'pytest' finds all tests within the `src/test` folder and runs them.
3. Packaging/Building - using the [setup.py](../setup.py) file, the application is packaged into a wheel (.whl) file.
#### Manual Step
Package publish is a manually triggered job.
To run it, navigate to the 'CI/CD Jobs' page of the Wellbore DDMS Data Loader GitLab project:
[https://community.opengroup.org/osdu/platform/data-flow/data-loading/wellbore-ddms-las-loader/-/jobs](https://community.opengroup.org/osdu/platform/data-flow/data-loading/wellbore-ddms-las-loader/-/jobs).
Run the publish job for the build you want to publish. It will build and upload `wbdutil` with a specific package version number. The package version number is the `__VERSION__` found in the `__init__.py` file for the module followed by a build tag. The build tag is based upon the CI `build_id`:
1. The default build tag is an alpha release of the form 1.2.3a456
1. If the repo branch has been tagged with a string starting with `BETA` then the beta release form is used 1.2.3b456.
1. If the repo branch has been tagged with a string starting with `RELEASE` then the primary release form is used 1.2.3.456.
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment