Skip to content
GitLab
Commit 4bc9dbec authored by Chad Leong's avatar Chad Leong
Browse files

Merge branch 'update/read-me' into 'main' 

remove development note

See merge request !42
parents 0fc6cfe2 96458af5
Pipeline #107561 passed with stages
in 1 minute and 55 seconds
Hide whitespace changes
Inline Side-by-side
README.md
View file @ 4bc9dbec
# Data Loader Utility for Wellbore DDMS # Data Loader Utility for Wellbore DDMS
## :exclamation: This is currently in development
## Description ## Description
1. This will be the OSDU Data loader utility to support Wellbore DDMS. 1. This will be the OSDU Data loader utility to support Wellbore DDMS.
2. This is a wrapper behind the Wellbore DDMS [API](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/blob/master/spec/generated/openapi.json) 2. This is a wrapper behind the Wellbore DDMS [API](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/blob/master/spec/generated/openapi.json)
## Install from Package Registry ## Install from Package Registry
### Before installation: ### Before installation:
- Ensure Python 3.8 or newer is installed and that Python has been added to the PATH (in your system's environment variables).
- Confirm you have `pip` installed by running the following command (it should be installed automatically when you installed Python) - `pip --version` - Ensure Python 3.8 or newer is installed and that Python has been added to the PATH (in your system's environment variables).
- If installing `wbdutil` within a virtual environment, first install `pipenv` using the following command - `pip install pipenv` - Confirm you have `pip` installed by running the following command (it should be installed automatically when you installed Python) - `pip --version`
- Note: There is a bug in `pipenv` version 2021.11.23 which will cause the `wbdutil` installation command to fail. Ensure your version number is different to this (e.g. `pip install pipenv=="2021.11.15"` or alternatively a newer version). - If installing `wbdutil` within a virtual environment, first install `pipenv` using the following command - `pip install pipenv`
- Note: There is a bug in `pipenv` version 2021.11.23 which will cause the `wbdutil` installation command to fail. Ensure your version number is different to this (e.g. `pip install pipenv=="2021.11.15"` or alternatively a newer version).
### Installation: ### Installation:
The simplest way to install `wbdutil` is from the community package registry. The simplest way to install `wbdutil` is from the community package registry.
**Method 1:** To download and install `wbdutil` on your machine run: **Method 1:** To download and install `wbdutil` on your machine run:
``` ```
pip install wbdutil --extra-index-url https://community.opengroup.org/api/v4/projects/801/packages/pypi/simple pip install wbdutil --extra-index-url https://community.opengroup.org/api/v4/projects/801/packages/pypi/simple
``` ```
**Method 2:** Alternatively, you may install the `wbdutil` package within a virtual environment using `pipenv`. If you have `pipenv` installed, simply run: **Method 2:** Alternatively, you may install the `wbdutil` package within a virtual environment using `pipenv`. If you have `pipenv` installed, simply run:
``` ```
pipenv install wbdutil --extra-index-url https://community.opengroup.org/api/v4/projects/801/packages/pypi/simple --skip-lock pipenv install wbdutil --extra-index-url https://community.opengroup.org/api/v4/projects/801/packages/pypi/simple --skip-lock
``` ```
## Usage ## Usage
The `wbdutil` package has a command line interface of the same name `wbdutil` which has the general syntax: The `wbdutil` package has a command line interface of the same name `wbdutil` which has the general syntax:
``` ```
wbdutil <group> <command> options wbdutil <group> <command> options
``` ```
...@@ -40,30 +42,31 @@ wbdutil <group> <command> options ...@@ -40,30 +42,31 @@ wbdutil <group> <command> options
Help for any group or command can be obtained by using the `-h` option. Help for any group or command can be obtained by using the `-h` option.
There are several groups: There are several groups:
* `download`: Download a welllog and curve data to a las format file.
* `ingest`: Upload a wellbore, welllog and/or bulk data to an OSDU instance - `download`: Download a welllog and curve data to a las format file.
* `list`: List the data held in OSDU for a given wellbore or welllog id. - `ingest`: Upload a wellbore, welllog and/or bulk data to an OSDU instance
* `parse`: Dry run of the ingest command, instead of uploading data to the OSDU instance it creates json files for the wellbore and welllog. - `list`: List the data held in OSDU for a given wellbore or welllog id.
* `search`: Search for a wellbore given a well name. - `parse`: Dry run of the ingest command, instead of uploading data to the OSDU instance it creates json files for the wellbore and welllog.
* `update`: Update the existing bulk data for a given welllog - `search`: Search for a wellbore given a well name.
- `update`: Update the existing bulk data for a given welllog
The `wbdutil` requires configuration information, read from a [configuration file](#config-file). The `wbdutil` requires configuration information, read from a [configuration file](#config-file).
The path to this file must either be provided in the environment variable `CONFIGPATH` or as a command line option. The path to this file must either be provided in the environment variable `CONFIGPATH` or as a command line option.
All the commands that connect to an OSDU instance (e.g. commands within the `ingest` group) require a bearer token. All the commands that connect to an OSDU instance (e.g. commands within the `ingest` group) require a bearer token.
This must be provided either in the environment variable `OSDUTOKEN` or as a command line option. This must be provided either in the environment variable `OSDUTOKEN` or as a command line option.
### Environment variables ### Environment variables
| Varaible name | overriding command option | Comment | | Varaible name | overriding command option | Comment |
| ------------- | --------------------------| ------- | | ------------- | ------------------------- | ---------------------------------------------------------- |
| OSDUTOKEN | `--token` `-t` | The JWT required to authenticate against an OSDU instance. | | OSDUTOKEN | `--token` `-t` | The JWT required to authenticate against an OSDU instance. |
| CONFIGPATH | `--config_path` `-c` | The path to the configuration file. | | CONFIGPATH | `--config_path` `-c` | The path to the configuration file. |
### Config file ### Config file
The `wbdutil` requires a configuration file that has the following JSON structure: The `wbdutil` requires a configuration file that has the following JSON structure:
``` ```
{ {
"base_url": "https://osdu-ship.msft-osdu-test.org", "base_url": "https://osdu-ship.msft-osdu-test.org",
...@@ -72,7 +75,7 @@ The `wbdutil` requires a configuration file that has the following JSON structur ...@@ -72,7 +75,7 @@ The `wbdutil` requires a configuration file that has the following JSON structur
{ {
"legaltags": ["opendes-public-usa-dataset-7643990"], "legaltags": ["opendes-public-usa-dataset-7643990"],
"otherRelevantDataCountries": ["US"], "otherRelevantDataCountries": ["US"],
"status": "compliant" "status": "compliant"
}, },
"data": { "data": {
"default": { "default": {
...@@ -85,7 +88,7 @@ The `wbdutil` requires a configuration file that has the following JSON structur ...@@ -85,7 +88,7 @@ The `wbdutil` requires a configuration file that has the following JSON structur
} }
``` ```
The `base_url` and `data_partition_id` must be correct for the OSDU instance that you want to connect to. The `base_url` and `data_partition_id` must be correct for the OSDU instance that you want to connect to.
`wellbore_mapping` and `welllog_mapping` are optional features described in the [custom mappings](#custom-mappings) section. `wellbore_mapping` and `welllog_mapping` are optional features described in the [custom mappings](#custom-mappings) section.
#### Custom mappings #### Custom mappings
...@@ -99,7 +102,7 @@ There are 3 mapping definitions `wellbore_mapping`, `welllog_mapping` and `las_f ...@@ -99,7 +102,7 @@ There are 3 mapping definitions `wellbore_mapping`, `welllog_mapping` and `las_f
The first 2 (`wellbore_mapping` and `welllog_mapping`) define mappings from LAS format data to OSDU wellbore and welllog objects. The first 2 (`wellbore_mapping` and `welllog_mapping`) define mappings from LAS format data to OSDU wellbore and welllog objects.
`las_file_mapping` defines the mapping from OSDU well log, well bore and curve data to LAS format data (a `lasio.LASFile` object). `las_file_mapping` defines the mapping from OSDU well log, well bore and curve data to LAS format data (a `lasio.LASFile` object).
All the mapping definitions must contain a `mapping` attribute, in addition the LAS to OSDU mapping definitions (`wellbore_mapping` and `welllog_mapping`) must contain a `kind` attribute. All the mapping definitions must contain a `mapping` attribute, in addition the LAS to OSDU mapping definitions (`wellbore_mapping` and `welllog_mapping`) must contain a `kind` attribute.
If `wellbore_mapping`, `welllog_mapping` and `las_file_mapping` are not defined in the configuration file `wbdutil` will use the default mappings. If `wellbore_mapping`, `welllog_mapping` and `las_file_mapping` are not defined in the configuration file `wbdutil` will use the default mappings.
The `mapping` attribute describes how data in the incoming object should be transformed into the outgoing data type. The `mapping` attribute describes how data in the incoming object should be transformed into the outgoing data type.
The `kind` attribute defines the target OSDU data type (kind), for example `osdu:wks:work-product-component--WellLog:1.1.0`. The `kind` attribute defines the target OSDU data type (kind), for example `osdu:wks:work-product-component--WellLog:1.1.0`.
...@@ -143,7 +146,7 @@ Here is an example mapping for a welllog that could be added to a configuration ...@@ -143,7 +146,7 @@ Here is an example mapping for a welllog that could be added to a configuration
} }
``` ```
The simple data mappings take the form of a key and string value pair. The simple data mappings take the form of a key and string value pair.
The key (string to the left of the semi-colon) is the target field within the OSDU data kind and The key (string to the left of the semi-colon) is the target field within the OSDU data kind and
the value string defines the source of the data. For example: the value string defines the source of the data. For example:
`"data.ReferenceCurveID": "curves[0].mnemonic"` will set the `data.ReferenceCurveID` field of the output OSDU object `"data.ReferenceCurveID": "curves[0].mnemonic"` will set the `data.ReferenceCurveID` field of the output OSDU object
...@@ -157,6 +160,7 @@ There are often more complex transformations that need to be performed on the in ...@@ -157,6 +160,7 @@ There are often more complex transformations that need to be performed on the in
`wbdutil` supports two types of complex mapping `array` and `function`. `wbdutil` supports two types of complex mapping `array` and `function`.
The `function` complex mapping type makes a call to a hard coded function to perform a transformation on the incoming data. The `function` complex mapping type makes a call to a hard coded function to perform a transformation on the incoming data.
For example: For example:
``` ```
"CurveUnit": { "CurveUnit": {
"type": "function", "type": "function",
...@@ -167,12 +171,14 @@ For example: ...@@ -167,12 +171,14 @@ For example:
] ]
} }
``` ```
This will set the value of the `CurveUnit` field to the output of the function `las2osdu_curve_uom_converter` using the input arguments in the args array. The `args` section defines the argument for the function, each `arg` is not the direct input argument to the function. An `arg` is a reference to piece of data in the incoming data or configuration file. This will set the value of the `CurveUnit` field to the output of the function `las2osdu_curve_uom_converter` using the input arguments in the args array. The `args` section defines the argument for the function, each `arg` is not the direct input argument to the function. An `arg` is a reference to piece of data in the incoming data or configuration file.
In this case `unit` references data in the input data and `data_partition_id` data in the configuration file. In this case `unit` references data in the input data and `data_partition_id` data in the configuration file.
The second complex mapping is `array` this should be used if the elements of an incoming array need to be changed in some way. The second complex mapping is `array` this should be used if the elements of an incoming array need to be changed in some way.
This could be a field name change, a change in the object structure or to call a function on specific data within each element. This could be a field name change, a change in the object structure or to call a function on specific data within each element.
Here is an example: Here is an example:
``` ```
{ {
"data.Curves": { "data.Curves": {
...@@ -193,15 +199,17 @@ Here is an example: ...@@ -193,15 +199,17 @@ Here is an example:
} }
} }
``` ```
This mapping will iterate over the `curves` array of the input `lasio.LASFile` object and apply an This mapping will iterate over the `curves` array of the input `lasio.LASFile` object and apply an
inner mapping to each element in the array. inner mapping to each element in the array.
In this case the inner mapping is defined so that the `mnemonic` field of the `curve` element is In this case the inner mapping is defined so that the `mnemonic` field of the `curve` element is
mapped to both the `CurveID` and `Mnemonic` output fields, and the `CurveUnit` output field is set mapped to both the `CurveID` and `Mnemonic` output fields, and the `CurveUnit` output field is set
to the return value of the function `las2osdu_curve_uom_converter` to the return value of the function `las2osdu_curve_uom_converter`
that takes the `unit` field of array element and the `data_partition_id` (from configuration) as arguments. that takes the `unit` field of array element and the `data_partition_id` (from configuration) as arguments.
The resulting output array is mapped to the `data.Curves` field of the output OSDU kind. The resulting output array is mapped to the `data.Curves` field of the output OSDU kind.
Here is an example `las_file_mapping` section: Here is an example `las_file_mapping` section:
``` ```
"las_file_mapping": { "las_file_mapping": {
"mapping": { "mapping": {
...@@ -219,6 +227,7 @@ Here is an example `las_file_mapping` section: ...@@ -219,6 +227,7 @@ Here is an example `las_file_mapping` section:
} }
} }
``` ```
With OSDU to LAS mappings, data is drawn from 3 types of OSDU data object: wellbore, welllog and curves. With OSDU to LAS mappings, data is drawn from 3 types of OSDU data object: wellbore, welllog and curves.
Each of these incoming OSDU data can be referenced by the keywords `WELLBORE`, `WELLLOG` and `CURVES`. Each of these incoming OSDU data can be referenced by the keywords `WELLBORE`, `WELLLOG` and `CURVES`.
Where `WELLBORE` and `WELLLOG` are wellbore and welllog OSDU kinds and `CURVES` is a Pandas DataFrame that Where `WELLBORE` and `WELLLOG` are wellbore and welllog OSDU kinds and `CURVES` is a Pandas DataFrame that
...@@ -229,21 +238,21 @@ it also contains example custom mappings for the `osdu:wks:master-data--Wellbore ...@@ -229,21 +238,21 @@ it also contains example custom mappings for the `osdu:wks:master-data--Wellbore
This table summarises the available keywords. This table summarises the available keywords.
| Keyword | Valid Mapping type | Incomming data source | | Keyword | Valid Mapping type | Incomming data source |
| ---------------------------------------- | -------------------| ------------------------- | | --------------- | ------------------ | ------------------------- |
| `CONFIGURATION` | All | The configuration file | | `CONFIGURATION` | All | The configuration file |
| `WELLBORE` | `las_file_mapping` | The OSDU wellbore object | | `WELLBORE` | `las_file_mapping` | The OSDU wellbore object |
| `WELLLOG` | `las_file_mapping` | The OSDU welllog object | | `WELLLOG` | `las_file_mapping` | The OSDU welllog object |
| `CURVES` | `las_file_mapping` | The OSDU Curves DataFrame | | `CURVES` | `las_file_mapping` | The OSDU Curves DataFrame |
There are a limited number of mapping functions available these are listed below: There are a limited number of mapping functions available these are listed below:
| Function name | Mapping type | Purpose | | Function name | Mapping type | Purpose |
| ---------------------------------------------------------- | -------------------| ----------------------------------------------------- | | ---------------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------- |
| `build_wellbore_name_aliases(uwi, data_partition_id)` | `wellbore_mapping` | Constructs a name alias object from the LAS UWI and the data partition id. | | `build_wellbore_name_aliases(uwi, data_partition_id)` | `wellbore_mapping` | Constructs a name alias object from the LAS UWI and the data partition id. |
| `get_wellbore_id()` | `welllog_mapping` | Returns the wellbore id from the wellbore that corresponds to the welllog | | `get_wellbore_id()` | `welllog_mapping` | Returns the wellbore id from the wellbore that corresponds to the welllog |
| `las2osdu_curve_uom_converter(unit, data_partition_id)` | `welllog_mapping` | This function converts a LAS format unit of measure to an OSDU format UoM. | | `las2osdu_curve_uom_converter(unit, data_partition_id)` | `welllog_mapping` | This function converts a LAS format unit of measure to an OSDU format UoM. |
| `extract_uwi_from_name_aliases(NameAliases: list)` | `las_file_mapping` | Return the first name alias or None if none exist | | `extract_uwi_from_name_aliases(NameAliases: list)` | `las_file_mapping` | Return the first name alias or None if none exist |
| `build_curves_section(wl_curves: list, curves: DataFrame)` | `las_file_mapping` | Iterates over curves, converting units of measure from OSDU to LAS form. Returns the updated curve data. | | `build_curves_section(wl_curves: list, curves: DataFrame)` | `las_file_mapping` | Iterates over curves, converting units of measure from OSDU to LAS form. Returns the updated curve data. |
These are hard coded functions, so a change request will need to be raised if additional functions are required. We have avoided user defined functions, because such functions represent a small security risk. These are hard coded functions, so a change request will need to be raised if additional functions are required. We have avoided user defined functions, because such functions represent a small security risk.
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment