Commit 0fc6cfe2 authored by Chad Leong's avatar Chad Leong
Browse files

Merge branch '47-soft-mapping-for-osdu-welllog-to-las-file' into 'main'

Soft mapping OSDU -> LasFileMappingFunctions

See merge request !41
parents bffd7c45 5ea6996a
Pipeline #101631 passed with stages
in 2 minutes and 1 second
......@@ -91,14 +91,18 @@ The `base_url` and `data_partition_id` must be correct for the OSDU instance tha
#### Custom mappings
**Custom mappings are an advanced feature of `wbdutil` that require knowledge of both `lasio.LASFile` and OSDU data object schemas and should be used with care.**
The configuration file can be used to define optional custom mappings from `lasio.LASFile` data objects
to OSDU wellbore and welllog objects of a specified kind.
The configuration file can be used to define optional custom mappings between `lasio.LASFile` data objects
and OSDU wellbore and welllog objects of a specified kind.
It is recommended that a new mapping is thoroughly tested using the `parse` command group, before upload to OSDU.
There are 2 mapping types `wellbore_mapping` and `welllog_mapping` both must contain a `kind` and a `mapping` attribute.
If `wellbore_mapping` or `welllog_mapping` are not defined in the configuration file `wbdutil` will use the default mappings for the well bore and/or well log.
There are 3 mapping definitions `wellbore_mapping`, `welllog_mapping` and `las_file_mapping`.
The first 2 (`wellbore_mapping` and `welllog_mapping`) define mappings from LAS format data to OSDU wellbore and welllog objects.
`las_file_mapping` defines the mapping from OSDU well log, well bore and curve data to LAS format data (a `lasio.LASFile` object).
All the mapping definitions must contain a `mapping` attribute, in addition the LAS to OSDU mapping definitions (`wellbore_mapping` and `welllog_mapping`) must contain a `kind` attribute.
If `wellbore_mapping`, `welllog_mapping` and `las_file_mapping` are not defined in the configuration file `wbdutil` will use the default mappings.
The `mapping` attribute describes how data in the incoming object should be transformed into the outgoing data type.
The `kind` attribute defines the target OSDU data type (kind), for example `osdu:wks:work-product-component--WellLog:1.1.0`.
The `mapping` type describes how data in the incoming `lasio.LASFile` object should be transformed into the target OSDU data kind.
Here is an example mapping for a welllog that could be added to a configuration file.
```
......@@ -163,7 +167,8 @@ For example:
]
}
```
This will set the value of the `CurveUnit` field to the output of the function `las2osdu_curve_uom_converter` using the input arguments in the args array (in this case `unit` from the input array element and `data_partition_id` from the configuration file).
This will set the value of the `CurveUnit` field to the output of the function `las2osdu_curve_uom_converter` using the input arguments in the args array. The `args` section defines the argument for the function, each `arg` is not the direct input argument to the function. An `arg` is a reference to piece of data in the incoming data or configuration file.
In this case `unit` references data in the input data and `data_partition_id` data in the configuration file.
The second complex mapping is `array` this should be used if the elements of an incoming array need to be changed in some way.
This could be a field name change, a change in the object structure or to call a function on specific data within each element.
......@@ -196,15 +201,49 @@ to the return value of the function `las2osdu_curve_uom_converter`
that takes the `unit` field of array element and the `data_partition_id` (from configuration) as arguments.
The resulting output array is mapped to the `data.Curves` field of the output OSDU kind.
Here is an example `las_file_mapping` section:
```
"las_file_mapping": {
"mapping": {
"Well.WELL": "WELLBORE.data.FacilityName",
"Well.UWI": {
"type": "function",
"function": "extract_uwi_from_name_aliases",
"args": ["WELLBORE.data.NameAliases"]
},
"Curves": {
"type": "function",
"function": "build_curves_section",
"args": ["WELLLOG.data.Curves", "CURVES"]
}
}
}
```
With OSDU to LAS mappings, data is drawn from 3 types of OSDU data object: wellbore, welllog and curves.
Each of these incoming OSDU data can be referenced by the keywords `WELLBORE`, `WELLLOG` and `CURVES`.
Where `WELLBORE` and `WELLLOG` are wellbore and welllog OSDU kinds and `CURVES` is a Pandas DataFrame that
contains the incoming curves data from OSDU.
An example configuration file that is setup for the preship OSDU instance is given in `src/example_opendes_configuration.json`,
it also contains example custom mappings for the `osdu:wks:master-data--Wellbore:1.0.0` wellbore kind and the `osdu:wks:work-product-component--WellLog:1.1.0` welllog kind.
This table summarises the available keywords.
| Keyword | Valid Mapping type | Incomming data source |
| ---------------------------------------- | -------------------| ------------------------- |
| `CONFIGURATION` | All | The configuration file |
| `WELLBORE` | `las_file_mapping` | The OSDU wellbore object |
| `WELLLOG` | `las_file_mapping` | The OSDU welllog object |
| `CURVES` | `las_file_mapping` | The OSDU Curves DataFrame |
There are a limited number of mapping functions available these are listed below:
| Function name | Mapping type | Purpose |
| ------------------------------------------------------ | -------------------| ----------------------------------------------------- |
| `build_wellbore_name_aliases(uwi, data_partition_id)` | `wellbore_mapping` | Constructs a name alias object from the LAS UWI and the data partition id. |
| `get_wellbore_id()` | `welllog_mapping` | Returns the wellbore id from the wellbore that corresponds to the welllog |
| `las2osdu_curve_uom_converter(unit, data_partition_id)`| `welllog_mapping` | This function converts a LAS format unit of measure to an OSDU format UoM. |
| Function name | Mapping type | Purpose |
| ---------------------------------------------------------- | -------------------| ----------------------------------------------------- |
| `build_wellbore_name_aliases(uwi, data_partition_id)` | `wellbore_mapping` | Constructs a name alias object from the LAS UWI and the data partition id. |
| `get_wellbore_id()` | `welllog_mapping` | Returns the wellbore id from the wellbore that corresponds to the welllog |
| `las2osdu_curve_uom_converter(unit, data_partition_id)` | `welllog_mapping` | This function converts a LAS format unit of measure to an OSDU format UoM. |
| `extract_uwi_from_name_aliases(NameAliases: list)` | `las_file_mapping` | Return the first name alias or None if none exist |
| `build_curves_section(wl_curves: list, curves: DataFrame)` | `las_file_mapping` | Iterates over curves, converting units of measure from OSDU to LAS form. Returns the updated curve data. |
These are hard coded functions, so a change request will need to be raised if additional functions are required. We have avoided user defined functions, because such functions represent a small security risk.
......@@ -60,5 +60,20 @@
}
}
}
}
},
"las_file_mapping": {
"mapping": {
"Well.WELL": "WELLBORE.data.FacilityName",
"Well.UWI": {
"type": "function",
"function": "extract_uwi_from_name_aliases",
"args": ["WELLBORE.data.NameAliases"]
},
"Curves": {
"type": "function",
"function": "build_curves_section",
"args": ["WELLLOG.data.Curves", "CURVES"]
}
}
}
}
......@@ -33,7 +33,7 @@ class TestPropertyMapper:
}
# Act
result = target.remap_data(source)
result = target.remap_data(source, {})
# Assert
assert result["xyz"] is source["abc"]
......@@ -61,7 +61,7 @@ class TestPropertyMapper:
}
# Act
result = target.remap_data(source)
result = target.remap_data(source, {})
# Assert
assert result["dest1"] is source["level1a"]["level2"]["level3"]
......@@ -90,7 +90,7 @@ class TestPropertyMapper:
}
# Act
result = target.remap_data(source)
result = target.remap_data(source, {})
# Assert
assert result["dest1"] is None
......@@ -112,7 +112,7 @@ class TestPropertyMapper:
}
# Act
result = target.remap_data(source)
result = target.remap_data(source, {})
# Assert
assert result["level1"]["level2"]["a"] is source["a"]
......@@ -131,7 +131,7 @@ class TestPropertyMapper:
}
# Act
result = target.remap_data(source)
result = target.remap_data(source, {})
# Assert
assert result["dest"] is source["src"][1]
......@@ -149,7 +149,7 @@ class TestPropertyMapper:
}
# Act
result = target.remap_data(source)
result = target.remap_data(source, {})
# Assert
assert result["dest"] is source["src"][0][1]
......@@ -167,7 +167,7 @@ class TestPropertyMapper:
}
# Act
result = target.remap_data(source)
result = target.remap_data(source, {})
# Assert
assert result["dest"] == 123
......@@ -186,7 +186,7 @@ class TestPropertyMapper:
# Act
with pytest.raises(WbdutilAttributeError) as ex:
target.remap_data(source)
target.remap_data(source, {})
assert "src[a]" in ex.value.message
......@@ -204,7 +204,7 @@ class TestPropertyMapper:
# Act
with pytest.raises(WbdutilAttributeError) as ex:
target.remap_data(source)
target.remap_data(source, {})
assert "[1]" in ex.value.message
......@@ -224,7 +224,7 @@ class TestPropertyMapper:
}
# Act
result = target.remap_data(source)
result = target.remap_data(source, {})
# Assert
assert result["dest"]["dest2"] == "A value"
......@@ -248,7 +248,7 @@ class TestPropertyMapper:
source = {}
# Act
result = target.remap_data(source)
result = target.remap_data(source, {})
# Assert
assert result["data"]["WellboreID"] == "12345"
......@@ -277,7 +277,7 @@ class TestPropertyMapper:
}
# Act
result = target.remap_data(source)
result = target.remap_data(source, {})
# Assert
assert result["data"]["function_result"] == "12345"
......@@ -302,7 +302,7 @@ class TestPropertyMapper:
# Act
# Assert
with pytest.raises(WbdutilAttributeError) as execinfo:
target.remap_data(source)
target.remap_data(source, {})
assert execinfo.value.message == "The function 'some_function_that_does_not_exist' was not found in the provided mapping functions"
......@@ -324,7 +324,7 @@ class TestPropertyMapper:
# Act
# Assert
with pytest.raises(WbdutilAttributeError) as execinfo:
target.remap_data({})
target.remap_data({}, {})
assert execinfo.value.message == "All of the mapping function arguments must be strings."
......@@ -348,7 +348,7 @@ class TestPropertyMapper:
}
# Act
result = target.remap_data(source)
result = target.remap_data(source, {})
# Assert
result_array = result["data"]["array_remap_result"]
......
......@@ -312,7 +312,9 @@ class TestMapWellLogToLas:
})
data = DataFrame({"ABC": [1, 2, 3], "XYZ": [9, 8, 7], "IJK": [-1, -2, -3]})
subject = MapWellLogToLas(wellbore, welllog, data)
mockConfig = Mock(spec=Configuration)
mockConfig.las_file_mapping = None
subject = MapWellLogToLas(mockConfig, wellbore, welllog, data)
# ACT
result = subject.build_las_file()
......@@ -353,7 +355,9 @@ class TestMapWellLogToLas:
data = DataFrame({})
subject = MapWellLogToLas(wellbore, welllog, data)
mockConfig = Mock(spec=Configuration)
mockConfig.las_file_mapping = None
subject = MapWellLogToLas(mockConfig, wellbore, welllog, data)
# ACT
result = subject.build_las_file()
......@@ -371,7 +375,10 @@ class TestMapWellLogToLas:
wellbore = Record({"kind": "BoreKind", "acl": {}, "legal": {}, "data": {}})
data = DataFrame({"ABC": [1, 2, 3], "XYZ": [9, 8, 7], "IJK": [-1, -2, -3]})
subject = MapWellLogToLas(wellbore, welllog, data)
mockConfig = Mock(spec=Configuration)
mockConfig.las_file_mapping = None
subject = MapWellLogToLas(mockConfig, wellbore, welllog, data)
# ACT
result = subject.build_las_file()
......
......@@ -20,6 +20,8 @@ LasToRecordMapper = record_mapper.LasToRecordMapper
Record = record_mapper.Record
WellLogRecord = record_mapper.WellLogRecord
Configuration = import_module(f"{TARGET_MODULE_NAME}.common.configuration").Configuration
class TestWellLogService:
......@@ -213,8 +215,11 @@ class TestWellLogService:
arg_welllog_id = "wellog_id argument"
arg_curves = ["ABC", "XYZ"]
mockConfig = Mock(spec=Configuration)
mockConfig.las_file_mapping = None
# Act
result = subject.download_and_construct_las(arg_welllog_id, arg_curves)
result = subject.download_and_construct_las(mockConfig, arg_welllog_id, arg_curves)
# Assert
client.get_welllog_record.assert_called_once_with(arg_welllog_id)
......@@ -258,8 +263,11 @@ class TestWellLogService:
arg_welllog_id = "wellog_id argument"
arg_curves = ["ABC", "XYZ"]
mockConfig = Mock(spec=Configuration)
mockConfig.las_file_mapping = None
# Act
result = subject.download_and_construct_las(arg_welllog_id, arg_curves)
result = subject.download_and_construct_las(mockConfig, arg_welllog_id, arg_curves)
# Assert
client.get_welllog_record.assert_called_once_with(arg_welllog_id)
......
......@@ -31,7 +31,7 @@ def download_las(welllog_id: str,
service = WellLogService(client)
las_file = service.download_and_construct_las(welllog_id, curves)
las_file = service.download_and_construct_las(config, welllog_id, curves)
logger.warning(f"Writing to file {outfile}")
......
from typing import Any, Dict, List, Union
from .file_loader import IFileLoader, JsonLoader
from .reflection_utilities import ReflectionHelper
class Configuration:
"""
Class to hold metadata for LAS ingestion
"""
def __init__(self, file_loader: IFileLoader, path: str) -> None:
"""
Load JSON configuration using the file loader and create a new instance of a Configuration
:param IFileLoader file_loader: The file loader instance to user
:param str path: The full path and filename of the configuration file.
"""
json_parser = JsonLoader(file_loader)
self._config = json_parser.load(path)
def get_recursive(self, qualified_attribute_name: List[str]) -> str:
"""
Gets an item specified by its qualified attribute name - eg ["legal", "legaltags"] will get config[legal][legaltags].
:param str qualified_attribute_name: The exception message
return: the base url for API calls
rtype: str
"""
return ReflectionHelper.getattr_recursive(self._config, qualified_attribute_name)
@property
def base_url(self) -> str:
"""
Gets the base url.
return: the base url for API calls
rtype: str
"""
return self._config.get("base_url")
@property
def data_partition_id(self) -> str:
"""
Gets the data partition Id.
return: the data partition id
rtype: str
"""
return self._config.get("data_partition_id")
@property
def wellbore_mapping(self) -> Union[Dict[str, Any], None]:
"""
Gets the wellbore_mapping.
return: the wellbore_mapping
rtype: Dict[str, Any]
"""
return self._config.get("wellbore_mapping")
@property
def welllog_mapping(self) -> Union[Dict[str, Any], None]:
"""
Gets the welllog_mapping.
return: the welllog_mapping
rtype: Dict[str, Any]
"""
return self._config.get("welllog_mapping")
from typing import Any, Dict, List, Union
from .file_loader import IFileLoader, JsonLoader
from .reflection_utilities import ReflectionHelper
class Configuration:
"""
Class to hold metadata for LAS ingestion
"""
def __init__(self, file_loader: IFileLoader, path: str) -> None:
"""
Load JSON configuration using the file loader and create a new instance of a Configuration
:param IFileLoader file_loader: The file loader instance to user
:param str path: The full path and filename of the configuration file.
"""
json_parser = JsonLoader(file_loader)
self._config = json_parser.load(path)
def get_recursive(self, qualified_attribute_name: List[str]) -> str:
"""
Gets an item specified by its qualified attribute name - eg ["legal", "legaltags"] will get config[legal][legaltags].
:param str qualified_attribute_name: The exception message
return: the base url for API calls
rtype: str
"""
return ReflectionHelper.getattr_recursive(self._config, qualified_attribute_name)
@property
def base_url(self) -> str:
"""
Gets the base url.
return: the base url for API calls
rtype: str
"""
return self._config.get("base_url")
@property
def data_partition_id(self) -> str:
"""
Gets the data partition Id.
return: the data partition id
rtype: str
"""
return self._config.get("data_partition_id")
@property
def wellbore_mapping(self) -> Union[Dict[str, Any], None]:
"""
Gets the wellbore_mapping.
return: the wellbore_mapping
rtype: Dict[str, Any]
"""
return self._config.get("wellbore_mapping")
@property
def welllog_mapping(self) -> Union[Dict[str, Any], None]:
"""
Gets the welllog_mapping.
return: the welllog_mapping
rtype: Dict[str, Any]
"""
return self._config.get("welllog_mapping")
@property
def las_file_mapping(self) -> Union[Dict[str, Any], None]:
"""
Gets the las_file_mapping.
return: the las_file_mapping
rtype: Dict[str, Any]
"""
return self._config.get("las_file_mapping")
from typing import List
class WbdutilException(Exception):
"""
Base class for exceptions raised within Wdutil
"""
def __init__(self, *args):
"""
Create a new instance of a WdutilException
:param *args object: The Exception arguments
"""
super().__init__(args)
class WbdutilValidationException(Exception):
"""
General validation exception raised within Wdutil
"""
def __init__(self, error_messages: List[str], *args):
"""
Create a new instance of a WdutilValidationException
:param error_messages List[str]: The validation errors
:param *args object: The Exception arguments
"""
self._error_messages = error_messages
super().__init__(args)
@property
def error_messages(self) -> List[str]:
"""
Gets the list of validation errors
return: the validation errors
rtype: List[str]
"""
return self._error_messages
class WbdutilAttributeError(WbdutilException):
"""
Common exception class for attribute errors raised in the Wbdutil
"""
def __init__(self, message: str, inner_exception: Exception, *args):
"""
Create a new instance of a ExtendedPropertiesAttributeError
:param str message: The exception message
:param inner_exception Exception: The inner exception
:param args object: The Exception arguments
"""
self.message = message
self.inner_exception = inner_exception
super().__init__(args)
class WbdutilValueError(Exception):
def __init__(self, *args):
"""
Create a new instance of a WbdutilValueError
:param *args object: The Exception arguments
"""
super().__init__(args)
from typing import List
class WbdutilException(Exception):
"""
Base class for exceptions raised within Wdutil
"""
def __init__(self, *args):
"""
Create a new instance of a WdutilException
:param *args object: The Exception arguments
"""
super().__init__(args)
class WbdutilValidationException(Exception):
"""
General validation exception raised within Wdutil
"""
def __init__(self, error_messages: List[str], *args):
"""
Create a new instance of a WdutilValidationException
:param error_messages List[str]: The validation errors
:param *args object: The Exception arguments
"""
self._error_messages = error_messages
super().__init__(args)
@property
def error_messages(self) -> List[str]:
"""
Gets the list of validation errors
return: the validation errors
rtype: List[str]
"""
return self._error_messages
class WbdutilAttributeError(WbdutilException):
"""
Common exception class for attribute errors raised in the Wbdutil
"""
def __init__(self, message: str, inner_exception: Exception, *args):
"""
Create a new instance of a ExtendedPropertiesAttributeError
:param str message: The exception message
:param inner_exception Exception: The inner exception
:param args object: The Exception arguments
"""
self.message = message
self.inner_exception = inner_exception
super().__init__(args)
class WbdutilValueError(Exception):
def __init__(self, *args):
"""
Create a new instance of a WbdutilValueError
:param *args object: The Exception arguments
"""
super().__init__(args)
import os
from pathlib import Path
from typing import Dict, List
import lasio
from lasio.las import LASFile
from knack.log import get_logger
from abc import ABC, abstractmethod
import json
logger = get_logger(__name__)
class IFileLoader(ABC):
@abstractmethod
def load(self, filepath: str) -> str:
"""
Open and load a file