Commit 0480c826 authored by Niall McDaid's avatar Niall McDaid
Browse files

Update README - rebase changes

parent 2b8281a6
Pipeline #97859 passed with stages
in 1 minute
# Data Loader utility for Wellbore DDMS
# Data Loader Utility for Wellbore DDMS
## :exclamation: This is currently in development
......@@ -68,7 +68,6 @@ The `wbdutil` requires a configuration file that has the following JSON structur
{
"base_url": "https://osdu-ship.msft-osdu-test.org",
"data_partition_id": "opendes",
"acl_domain": "contoso.com",
"legal":
{
"legaltags": ["opendes-public-usa-dataset-7643990"],
......@@ -80,8 +79,132 @@ The `wbdutil` requires a configuration file that has the following JSON structur
"viewers": [ "data.default.viewers@opendes.contoso.com" ],
"owners": [ "data.default.owners@opendes.contoso.com" ]
}
},
"wellbore_mapping": {},
"welllog_mapping": {}
}
```
The `base_url` and `data_partition_id` must be correct for the OSDU instance that you want to connect to.
`wellbore_mapping` and `welllog_mapping` are optional features described in the [custom mappings](#custom-mappings) section.
#### Custom mappings
**Custom mappings are an advanced feature of `wbdutil` that require knowledge of both `lasio.LASFile` and OSDU data object schemas and should be used with care.**
The configuration file can be used to define optional custom mappings from `lasio.LASFile` data objects
to OSDU wellbore and welllog objects of a specified kind.
It is recommended that a new mapping is thoroughly tested using the `parse` command group, before upload to OSDU.
There are 2 mapping types `wellbore_mapping` and `welllog_mapping` both must contain a `kind` and a `mapping` attribute.
If `wellbore_mapping` or `welllog_mapping` are not defined in the configuration file `wbdutil` will use the default mappings for the well bore and/or well log.
The `kind` attribute defines the target OSDU data type (kind), for example `osdu:wks:work-product-component--WellLog:1.1.0`.
The `mapping` type describes how data in the incoming `lasio.LASFile` object should be transformed into the target OSDU data kind.
Here is an example mapping for a welllog that could be added to a configuration file.
```
{
"welllog_mapping": {
"kind": "osdu:wks:work-product-component--WellLog:1.1.0",
"mapping":
{
"acl.viewers": "CONFIGURATION.data.default.viewers",
"acl.owners": "CONFIGURATION.data.default.owners",
"legal.legaltags": "CONFIGURATION.legal.legaltags",
"legal.otherRelevantDataCountries": "CONFIGURATION.legal.otherRelevantDataCountries",
"legal.status": "CONFIGURATION.legal.status",
"data.ReferenceCurveID": "curves[0].mnemonic",
"data.WellboreID": {
"type": "function",
"function": "get_wellbore_id",
"args": []
},
"data.Curves": {
"type": "array",
"source": "curves",
"mapping": {
"CurveID": "mnemonic",
"Mnemonic": "mnemonic",
"CurveUnit": {
"type": "function",
"function": "las2osdu_curve_uom_converter",
"args": [
"unit",
"CONFIGURATION.data_partition_id"
]
}
}
}
}
}
}
```
The `base_url`, `data_partition_id` and `acl_domain` must be correct for the OSDU instance you want to connect to. An example configuration file that is setup for the pre-ship OSDU instance is given in [src/example_opendes_configuration.json](src/example_opendes_configuration.json).
The simple data mappings take the form of a key and string value pair.
The key (string to the left of the semi-colon) is the target field within the OSDU data kind and
the value string defines the source of the data. For example:
`"data.ReferenceCurveID": "curves[0].mnemonic"` will set the `data.ReferenceCurveID` field of the output OSDU object
to the value of the `mnemonic` field of the first element in the `curves` array of the input `lasio.LASFile` object.
The `CONFIGURATION` keyword indicates that data should be taken from the configuration file, for example:
`"acl.viewers": "CONFIGURATION.data.default.viewers"` will set the `acl.viewers` field of the output OSDU object
to the value of the `data.default.viewers` field of the configuration. The simple mapping form supports the direct copying of
all objects including arrays from the incoming LAS data to the output OSDU data kind.
There are often more complex transformations that need to be performed on the incoming data,
`wbdutil` supports two types of complex mapping `array` and `function`.
The `function` complex mapping type makes a call to a hard coded function to perform a transformation on the incoming data.
For example:
```
"CurveUnit": {
"type": "function",
"function": "las2osdu_curve_uom_converter",
"args": [
"unit",
"CONFIGURATION.data_partition_id"
]
}
```
This will set the value of the `CurveUnit` field to the output of the function `las2osdu_curve_uom_converter` using the input arguments in the args array (in this case `unit` from the input array element and `data_partition_id` from the configuration file).
The second complex mapping is `array` this should be used if the elements of an incoming array need to be changed in some way.
This could be a field name change, a change in the object structure or to call a function on specific data within each element.
Here is an example:
```
{
"data.Curves": {
"type": "array",
"source": "curves",
"mapping": {
"CurveID": "mnemonic",
"Mnemonic": "mnemonic",
"CurveUnit": {
"type": "function",
"function": "las2osdu_curve_uom_converter",
"args": [
"unit",
"CONFIGURATION.data_partition_id"
]
}
}
}
}
```
This mapping will iterate over the `curves` array of the input `lasio.LASFile` object and apply an
inner mapping to each element in the array.
In this case the inner mapping is defined so that the `mnemonic` field of the `curve` element is
mapped to both the `CurveID` and `Mnemonic` output fields, and the `CurveUnit` output field is set
to the return value of the function `las2osdu_curve_uom_converter`
that takes the `unit` field of array element and the `data_partition_id` (from configuration) as arguments.
The resulting output array is mapped to the `data.Curves` field of the output OSDU kind.
An example configuration file that is setup for the preship OSDU instance is given in `src/example_opendes_configuration.json`,
it also contains example custom mappings for the `osdu:wks:master-data--Wellbore:1.0.0` wellbore kind and the `osdu:wks:work-product-component--WellLog:1.1.0` welllog kind.
There are a limited number of mapping functions available these are listed below:
| Function name | Mapping type | Purpose |
| ------------------------------------------------------ | -------------------| ----------------------------------------------------- |
| `build_wellbore_name_aliases(uwi, data_partition_id)` | `wellbore_mapping` | Constructs a name alias object from the LAS UWI and the data partition id. |
| `get_wellbore_id()` | `welllog_mapping` | Returns the wellbore id from the wellbore that corresponds to the welllog |
| `las2osdu_curve_uom_converter(unit, data_partition_id)`| `welllog_mapping` | This function converts a LAS format unit of measure to an OSDU format UoM. |
These are hard coded functions, so a change request will need to be raised if additional functions are required. We have avoided user defined functions, because such functions represent a small security risk.
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment