Commit 4bc9dbec authored by Chad Leong's avatar Chad Leong
Browse files

Merge branch 'update/read-me' into 'main'

remove development note

See merge request !42
parents 0fc6cfe2 96458af5
Pipeline #107561 passed with stages
in 1 minute and 55 seconds
# Data Loader Utility for Wellbore DDMS
## :exclamation: This is currently in development
## Description
1. This will be the OSDU Data loader utility to support Wellbore DDMS.
2. This is a wrapper behind the Wellbore DDMS [API](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/blob/master/spec/generated/openapi.json)
## Install from Package Registry
### Before installation:
- Ensure Python 3.8 or newer is installed and that Python has been added to the PATH (in your system's environment variables).
- Confirm you have `pip` installed by running the following command (it should be installed automatically when you installed Python) - `pip --version`
- If installing `wbdutil` within a virtual environment, first install `pipenv` using the following command - `pip install pipenv`
- Note: There is a bug in `pipenv` version 2021.11.23 which will cause the `wbdutil` installation command to fail. Ensure your version number is different to this (e.g. `pip install pipenv=="2021.11.15"` or alternatively a newer version).
- Ensure Python 3.8 or newer is installed and that Python has been added to the PATH (in your system's environment variables).
- Confirm you have `pip` installed by running the following command (it should be installed automatically when you installed Python) - `pip --version`
- If installing `wbdutil` within a virtual environment, first install `pipenv` using the following command - `pip install pipenv`
- Note: There is a bug in `pipenv` version 2021.11.23 which will cause the `wbdutil` installation command to fail. Ensure your version number is different to this (e.g. `pip install pipenv=="2021.11.15"` or alternatively a newer version).
### Installation:
The simplest way to install `wbdutil` is from the community package registry.
**Method 1:** To download and install `wbdutil` on your machine run:
```
pip install wbdutil --extra-index-url https://community.opengroup.org/api/v4/projects/801/packages/pypi/simple
```
**Method 2:** Alternatively, you may install the `wbdutil` package within a virtual environment using `pipenv`. If you have `pipenv` installed, simply run:
```
pipenv install wbdutil --extra-index-url https://community.opengroup.org/api/v4/projects/801/packages/pypi/simple --skip-lock
```
## Usage
The `wbdutil` package has a command line interface of the same name `wbdutil` which has the general syntax:
```
wbdutil <group> <command> options
```
......@@ -40,30 +42,31 @@ wbdutil <group> <command> options
Help for any group or command can be obtained by using the `-h` option.
There are several groups:
* `download`: Download a welllog and curve data to a las format file.
* `ingest`: Upload a wellbore, welllog and/or bulk data to an OSDU instance
* `list`: List the data held in OSDU for a given wellbore or welllog id.
* `parse`: Dry run of the ingest command, instead of uploading data to the OSDU instance it creates json files for the wellbore and welllog.
* `search`: Search for a wellbore given a well name.
* `update`: Update the existing bulk data for a given welllog
- `download`: Download a welllog and curve data to a las format file.
- `ingest`: Upload a wellbore, welllog and/or bulk data to an OSDU instance
- `list`: List the data held in OSDU for a given wellbore or welllog id.
- `parse`: Dry run of the ingest command, instead of uploading data to the OSDU instance it creates json files for the wellbore and welllog.
- `search`: Search for a wellbore given a well name.
- `update`: Update the existing bulk data for a given welllog
The `wbdutil` requires configuration information, read from a [configuration file](#config-file).
The path to this file must either be provided in the environment variable `CONFIGPATH` or as a command line option.
All the commands that connect to an OSDU instance (e.g. commands within the `ingest` group) require a bearer token.
All the commands that connect to an OSDU instance (e.g. commands within the `ingest` group) require a bearer token.
This must be provided either in the environment variable `OSDUTOKEN` or as a command line option.
### Environment variables
| Varaible name | overriding command option | Comment |
| ------------- | --------------------------| ------- |
| Varaible name | overriding command option | Comment |
| ------------- | ------------------------- | ---------------------------------------------------------- |
| OSDUTOKEN | `--token` `-t` | The JWT required to authenticate against an OSDU instance. |
| CONFIGPATH | `--config_path` `-c` | The path to the configuration file. |
| CONFIGPATH | `--config_path` `-c` | The path to the configuration file. |
### Config file
The `wbdutil` requires a configuration file that has the following JSON structure:
```
{
"base_url": "https://osdu-ship.msft-osdu-test.org",
......@@ -72,7 +75,7 @@ The `wbdutil` requires a configuration file that has the following JSON structur
{
"legaltags": ["opendes-public-usa-dataset-7643990"],
"otherRelevantDataCountries": ["US"],
"status": "compliant"
"status": "compliant"
},
"data": {
"default": {
......@@ -85,7 +88,7 @@ The `wbdutil` requires a configuration file that has the following JSON structur
}
```
The `base_url` and `data_partition_id` must be correct for the OSDU instance that you want to connect to.
The `base_url` and `data_partition_id` must be correct for the OSDU instance that you want to connect to.
`wellbore_mapping` and `welllog_mapping` are optional features described in the [custom mappings](#custom-mappings) section.
#### Custom mappings
......@@ -99,7 +102,7 @@ There are 3 mapping definitions `wellbore_mapping`, `welllog_mapping` and `las_f
The first 2 (`wellbore_mapping` and `welllog_mapping`) define mappings from LAS format data to OSDU wellbore and welllog objects.
`las_file_mapping` defines the mapping from OSDU well log, well bore and curve data to LAS format data (a `lasio.LASFile` object).
All the mapping definitions must contain a `mapping` attribute, in addition the LAS to OSDU mapping definitions (`wellbore_mapping` and `welllog_mapping`) must contain a `kind` attribute.
All the mapping definitions must contain a `mapping` attribute, in addition the LAS to OSDU mapping definitions (`wellbore_mapping` and `welllog_mapping`) must contain a `kind` attribute.
If `wellbore_mapping`, `welllog_mapping` and `las_file_mapping` are not defined in the configuration file `wbdutil` will use the default mappings.
The `mapping` attribute describes how data in the incoming object should be transformed into the outgoing data type.
The `kind` attribute defines the target OSDU data type (kind), for example `osdu:wks:work-product-component--WellLog:1.1.0`.
......@@ -143,7 +146,7 @@ Here is an example mapping for a welllog that could be added to a configuration
}
```
The simple data mappings take the form of a key and string value pair.
The simple data mappings take the form of a key and string value pair.
The key (string to the left of the semi-colon) is the target field within the OSDU data kind and
the value string defines the source of the data. For example:
`"data.ReferenceCurveID": "curves[0].mnemonic"` will set the `data.ReferenceCurveID` field of the output OSDU object
......@@ -157,6 +160,7 @@ There are often more complex transformations that need to be performed on the in
`wbdutil` supports two types of complex mapping `array` and `function`.
The `function` complex mapping type makes a call to a hard coded function to perform a transformation on the incoming data.
For example:
```
"CurveUnit": {
"type": "function",
......@@ -167,12 +171,14 @@ For example:
]
}
```
This will set the value of the `CurveUnit` field to the output of the function `las2osdu_curve_uom_converter` using the input arguments in the args array. The `args` section defines the argument for the function, each `arg` is not the direct input argument to the function. An `arg` is a reference to piece of data in the incoming data or configuration file.
In this case `unit` references data in the input data and `data_partition_id` data in the configuration file.
In this case `unit` references data in the input data and `data_partition_id` data in the configuration file.
The second complex mapping is `array` this should be used if the elements of an incoming array need to be changed in some way.
This could be a field name change, a change in the object structure or to call a function on specific data within each element.
Here is an example:
```
{
"data.Curves": {
......@@ -193,15 +199,17 @@ Here is an example:
}
}
```
This mapping will iterate over the `curves` array of the input `lasio.LASFile` object and apply an
inner mapping to each element in the array.
In this case the inner mapping is defined so that the `mnemonic` field of the `curve` element is
mapped to both the `CurveID` and `Mnemonic` output fields, and the `CurveUnit` output field is set
to the return value of the function `las2osdu_curve_uom_converter`
to the return value of the function `las2osdu_curve_uom_converter`
that takes the `unit` field of array element and the `data_partition_id` (from configuration) as arguments.
The resulting output array is mapped to the `data.Curves` field of the output OSDU kind.
Here is an example `las_file_mapping` section:
```
"las_file_mapping": {
"mapping": {
......@@ -219,6 +227,7 @@ Here is an example `las_file_mapping` section:
}
}
```
With OSDU to LAS mappings, data is drawn from 3 types of OSDU data object: wellbore, welllog and curves.
Each of these incoming OSDU data can be referenced by the keywords `WELLBORE`, `WELLLOG` and `CURVES`.
Where `WELLBORE` and `WELLLOG` are wellbore and welllog OSDU kinds and `CURVES` is a Pandas DataFrame that
......@@ -229,21 +238,21 @@ it also contains example custom mappings for the `osdu:wks:master-data--Wellbore
This table summarises the available keywords.
| Keyword | Valid Mapping type | Incomming data source |
| ---------------------------------------- | -------------------| ------------------------- |
| `CONFIGURATION` | All | The configuration file |
| `WELLBORE` | `las_file_mapping` | The OSDU wellbore object |
| `WELLLOG` | `las_file_mapping` | The OSDU welllog object |
| `CURVES` | `las_file_mapping` | The OSDU Curves DataFrame |
| Keyword | Valid Mapping type | Incomming data source |
| --------------- | ------------------ | ------------------------- |
| `CONFIGURATION` | All | The configuration file |
| `WELLBORE` | `las_file_mapping` | The OSDU wellbore object |
| `WELLLOG` | `las_file_mapping` | The OSDU welllog object |
| `CURVES` | `las_file_mapping` | The OSDU Curves DataFrame |
There are a limited number of mapping functions available these are listed below:
| Function name | Mapping type | Purpose |
| ---------------------------------------------------------- | -------------------| ----------------------------------------------------- |
| `build_wellbore_name_aliases(uwi, data_partition_id)` | `wellbore_mapping` | Constructs a name alias object from the LAS UWI and the data partition id. |
| `get_wellbore_id()` | `welllog_mapping` | Returns the wellbore id from the wellbore that corresponds to the welllog |
| `las2osdu_curve_uom_converter(unit, data_partition_id)` | `welllog_mapping` | This function converts a LAS format unit of measure to an OSDU format UoM. |
| `extract_uwi_from_name_aliases(NameAliases: list)` | `las_file_mapping` | Return the first name alias or None if none exist |
| Function name | Mapping type | Purpose |
| ---------------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------- |
| `build_wellbore_name_aliases(uwi, data_partition_id)` | `wellbore_mapping` | Constructs a name alias object from the LAS UWI and the data partition id. |
| `get_wellbore_id()` | `welllog_mapping` | Returns the wellbore id from the wellbore that corresponds to the welllog |
| `las2osdu_curve_uom_converter(unit, data_partition_id)` | `welllog_mapping` | This function converts a LAS format unit of measure to an OSDU format UoM. |
| `extract_uwi_from_name_aliases(NameAliases: list)` | `las_file_mapping` | Return the first name alias or None if none exist |
| `build_curves_section(wl_curves: list, curves: DataFrame)` | `las_file_mapping` | Iterates over curves, converting units of measure from OSDU to LAS form. Returns the updated curve data. |
These are hard coded functions, so a change request will need to be raised if additional functions are required. We have avoided user defined functions, because such functions represent a small security risk.
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment