Commit 9516364e authored by Gregory Harris's avatar Gregory Harris
Browse files

Rebase with squash.

Add a basic configurable remapping capability.

Flake8 files

Adding validation exception

Rename mapper

Reverse the mapping definition

Adding support for getting data from Configuration on mapping.

Capability to perform some complex remappings performing function and array operation.

Moving methods into the object_manipulation module from the property mapping module.

rename object_manipulation module to reflection_utilities

Moving ObjectManipulationAttributeError to the exceptions module and renaming to WbdutilAttributeError.

Exception renames

Removing ExtensibleEntity. Switching to dictionary rather than using the extendable object.

Add a test to illustrate a welllog remapping.

Implement default soft mapping for wellbore.

Removing AttributeBuilder

Fixing test

flake8 fixes

Fix the example mappings

Adding the welllog and wellbore mappings to the Configuration

Refactor the Record class so that it simply wraps a Dict

Bugfixes and renaming ObjectManipulation to ReflectionHelper.

Add some tests for reflection utilities and tidy up.

Updating the readme to cover custom data mappings.
parent 14d14604
Pipeline #96933 passed with stages
in 1 minute and 12 seconds
......@@ -82,7 +82,117 @@ The wbdutil requires a configuration file that has the following JSON structure:
}
```
The `base_url`, `data_partition_id` and `acl_domain` must be correct for the OSDU instance you want to connect to. An example configuration file that is setup for the preship OSDU instance is given in `src/example_opendes_configuration.json`.
The `base_url`, `data_partition_id` and `acl_domain` must be correct for the OSDU instance you want to connect to.
#### Custom mappings
The configuration file can be used to define custom mappings from `lasio.LASFile` data objects
to OSDU wellbore and welllog objects of a specified kind.
This is an advanced feature of `wbdutil` that requires knowledge of both `lasio.LASFile` and OSDU data object schemas and should be used with care.
It is recommended that a new mapping is thoroughly tested using the `parse` command group, before upload to OSDU.
There are 2 mapping types `wellbore_mapping` and `welllog_mapping` both must contain a `kind` and a `mapping` attribute.
The `kind` attribute defines the target OSDU data type (kind), for example `osdu:wks:work-product-component--WellLog:1.1.0`.
The `mapping` type describes how data in the incoming `lasio.LASFile` object should be transformed into the target OSDU data kind.
Here is an example.
```
{
"welllog_mapping": {
"kind": "osdu:wks:work-product-component--WellLog:1.1.0",
"mapping":
{
"acl.viewers": "CONFIGURATION.data.default.viewers",
"acl.owners": "CONFIGURATION.data.default.owners",
"legal.legaltags": "CONFIGURATION.legal.legaltags",
"legal.otherRelevantDataCountries": "CONFIGURATION.legal.otherRelevantDataCountries",
"legal.status": "CONFIGURATION.legal.status",
"data.ReferenceCurveID": "curves[0].mnemonic",
"data.WellboreID": {
"type": "function",
"function": "get_wellbore_id",
"args": []
},
"data.Curves": {
"type": "array",
"source": "curves",
"mapping": {
"CurveID": "mnemonic",
"Mnemonic": "mnemonic",
"CurveUnit": {
"type": "function",
"function": "las2osdu_curve_uom_converter",
"args": [
"unit",
"CONFIGURATION.data_partition_id"
]
}
}
}
}
}
}
```
The simple data mappings take the for of a key and string value pair.
The key (string to the left of the semi-colon) is the target field within the OSDU data kind and
the value string defines the source of the data. For example:
`"data.ReferenceCurveID": "curves[0].mnemonic"` will set the `data.ReferenceCurveID` field of the output OSDU object
to the value of the `mnemonic` field of the first element in the `curves` array of the input `lasio.LASFile` object.
The `CONFIGURATION` keyword indicates that data should be taken from the configuration file, for example:
`"acl.viewers": "CONFIGURATION.data.default.viewers"` will set the `acl.viewers` field of the output OSDU object
to the value of the `data.default.viewers` field of the configuration. The simple mapping form supports the direct copying of
all objects including arrays from the incoming LAS data to the output OSDU data kind.
There are often more complex transformations that need to be performed on the incoming data,
`wbdutil` supports two types of complex mapping `array` and `function`.
The `function` complex mapping type makes a call to a hard coded function to perform a transformation on the incoming data.
For example:
```
"CurveUnit": {
"type": "function",
"function": "las2osdu_curve_uom_converter",
"args": [
"unit",
"CONFIGURATION.data_partition_id"
]
}
```
This will set the value of the `CurveUnit` field to the output of the function `las2osdu_curve_uom_converter` using the input arguments in the args array (in this case `unit` from the input array element and `data_partition_id` from the configuration file).
The second complex mapping is `array` this should be used if the elements of an incoming array need to be changed in some way.
This could be a field name change, a change in the object structure or to call a function on specific data withing each element.
Here is an example:
```
{
"data.Curves": {
"type": "array",
"source": "curves",
"mapping": {
"CurveID": "mnemonic",
"Mnemonic": "mnemonic",
"CurveUnit": {
"type": "function",
"function": "las2osdu_curve_uom_converter",
"args": [
"unit",
"CONFIGURATION.data_partition_id"
]
}
}
}
}
```
This mapping will iterate over the `curves` array of the input `lasio.LASFile` object and apply an
inner mapping to each element in the array.
In this case the inner mapping is defined so that the `mnemonic` field of the `curve` element is
mapped to both the `CurveID` and `Mnemonic` output fields, and the `CurveUnit` output field is set
to the return value of the function `las2osdu_curve_uom_converter`
that takes the `unit` field of array element and the `data_partition_id` (from configuration) as arguments.
The resulting output array is mapped to the `data.Curves` field of the output OSDU kind.
An example configuration file that is setup for the preship OSDU instance is given in `src/example_opendes_configuration.json`,
it also contains example custom mappings for the `osdu:wks:master-data--Wellbore:1.0.0` wellbore kind and the `osdu:wks:work-product-component--WellLog:1.1.0` welllog kind.
## Development
......
......@@ -18,18 +18,19 @@
"default": {
"anyio": {
"hashes": [
"sha256:24adc69309fb5779bc1e06158e143e0b6d2c56b302a3ac3de3083c705a6ed39d",
"sha256:2855a9423524abcdd652d942f8932fda1735210f77a6b392eafd9ff34d3fe020"
"sha256:a0aeffe2fb1fdf374a8e4b471444f0f3ac4fb9f5a5b542b48824475e0042a5a6",
"sha256:b5fa16c5ff93fa1046f2eeb5bbff2dad4d3514d6cda61d02816dba34fa8c3c2e"
],
"markers": "python_full_version >= '3.6.2'",
"version": "==3.4.0"
"version": "==3.5.0"
},
"argcomplete": {
"hashes": [
"sha256:291f0beca7fd49ce285d2f10e4c1c77e9460cf823eef2de54df0c0fec88b0d81",
"sha256:2c7dbffd8c045ea534921e63b0be6fe65e88599990d8dc408ac8c542b72a5445"
"sha256:6372ad78c89d662035101418ae253668445b391755cfe94ea52f1b9d22425b20",
"sha256:cffa11ea77999bb0dd27bb25ff6dc142a6796142f68d45b1a26b11f58724561e"
],
"version": "==1.12.3"
"markers": "python_version >= '3.6'",
"version": "==2.0.0"
},
"certifi": {
"hashes": [
......@@ -40,11 +41,11 @@
},
"charset-normalizer": {
"hashes": [
"sha256:735e240d9a8506778cd7a453d97e817e536bb1fc29f4f6961ce297b9c7a917b0",
"sha256:83fcdeb225499d6344c8f7f34684c2981270beacc32ede2e669e94f7fa544405"
"sha256:876d180e9d7432c5d1dfd4c5d26b72f099d503e8fcc0feb7532c9289be60fcbd",
"sha256:cb957888737fc0bbcd78e3df769addb41fd1ff8cf950dc9e7ad7793f1bf44455"
],
"markers": "python_full_version >= '3.5.0'",
"version": "==2.0.8"
"version": "==2.0.10"
},
"colorama": {
"hashes": [
......@@ -64,19 +65,19 @@
},
"httpcore": {
"hashes": [
"sha256:9a98d2416b78976fc5396ff1f6b26ae9885efbb3105d24eed490f20ab4c95ec1",
"sha256:d10162a63265a0228d5807964bd964478cbdb5178f9a2eedfebb2faba27eef5d"
"sha256:9410fe352bea732311f2b2bee0555c8cc5e62b9a73b9d3272fe125a2aa6eb28e",
"sha256:d4305811f604d3c2e22869147392f134796976ff946c96a8cfba87f4e0171d83"
],
"markers": "python_version >= '3.6'",
"version": "==0.14.3"
"version": "==0.14.4"
},
"httpx": {
"hashes": [
"sha256:02af20df486b78892a614a7ccd4e4e86a5409ec4981ab0e422c579a887acad83",
"sha256:208e5ef2ad4d105213463cfd541898ed9d11851b346473539a8425e644bb7c66"
"sha256:7a3eb67ef0b8abbd6d9402248ef2f84a76080fa1c839f8662e6eb385640e445a",
"sha256:df9a0fd43fa79dbab411d83eb1ea6f7a525c96ad92e60c2d7f40388971b25777"
],
"index": "pypi",
"version": "==0.21.1"
"version": "==0.21.3"
},
"idna": {
"hashes": [
......@@ -111,69 +112,62 @@
},
"numpy": {
"hashes": [
"sha256:0b78ecfa070460104934e2caf51694ccd00f37d5e5dbe76f021b1b0b0d221823",
"sha256:1247ef28387b7bb7f21caf2dbe4767f4f4175df44d30604d42ad9bd701ebb31f",
"sha256:1403b4e2181fc72664737d848b60e65150f272fe5a1c1cbc16145ed43884065a",
"sha256:170b2a0805c6891ca78c1d96ee72e4c3ed1ae0a992c75444b6ab20ff038ba2cd",
"sha256:2e4ed57f45f0aa38beca2a03b6532e70e548faf2debbeb3291cfc9b315d9be8f",
"sha256:32fe5b12061f6446adcbb32cf4060a14741f9c21e15aaee59a207b6ce6423469",
"sha256:34f3456f530ae8b44231c63082c8899fe9c983fd9b108c997c4b1c8c2d435333",
"sha256:4c9c23158b87ed0e70d9a50c67e5c0b3f75bcf2581a8e34668d4e9d7474d76c6",
"sha256:5d95668e727c75b3f5088ec7700e260f90ec83f488e4c0aaccb941148b2cd377",
"sha256:615d4e328af7204c13ae3d4df7615a13ff60a49cb0d9106fde07f541207883ca",
"sha256:69077388c5a4b997442b843dbdc3a85b420fb693ec8e33020bb24d647c164fa5",
"sha256:74b85a17528ca60cf98381a5e779fc0264b4a88b46025e6bcbe9621f46bb3e63",
"sha256:81225e58ef5fce7f1d80399575576fc5febec79a8a2742e8ef86d7b03beef49f",
"sha256:8890b3360f345e8360133bc078d2dacc2843b6ee6059b568781b15b97acbe39f",
"sha256:92aafa03da8658609f59f18722b88f0a73a249101169e28415b4fa148caf7e41",
"sha256:9864424631775b0c052f3bd98bc2712d131b3e2cd95d1c0c68b91709170890b0",
"sha256:9e6f5f50d1eff2f2f752b3089a118aee1ea0da63d56c44f3865681009b0af162",
"sha256:a3deb31bc84f2b42584b8c4001c85d1934dbfb4030827110bc36bfd11509b7bf",
"sha256:ad010846cdffe7ec27e3f933397f8a8d6c801a48634f419e3d075db27acf5880",
"sha256:b1e2312f5b8843a3e4e8224b2b48fe16119617b8fc0a54df8f50098721b5bed2",
"sha256:bc988afcea53e6156546e5b2885b7efab089570783d9d82caf1cfd323b0bb3dd",
"sha256:c449eb870616a7b62e097982c622d2577b3dbc800aaf8689254ec6e0197cbf1e",
"sha256:c74c699b122918a6c4611285cc2cad4a3aafdb135c22a16ec483340ef97d573c",
"sha256:c885bfc07f77e8fee3dc879152ba993732601f1f11de248d4f357f0ffea6a6d4",
"sha256:e3c3e990274444031482a31280bf48674441e0a5b55ddb168f3a6db3e0c38ec8",
"sha256:e4799be6a2d7d3c33699a6f77201836ac975b2e1b98c2a07f66a38f499cb50ce",
"sha256:e6c76a87633aa3fa16614b61ccedfae45b91df2767cf097aa9c933932a7ed1e0",
"sha256:e89717274b41ebd568cd7943fc9418eeb49b1785b66031bc8a7f6300463c5898",
"sha256:f5162ec777ba7138906c9c274353ece5603646c6965570d82905546579573f73",
"sha256:fde96af889262e85aa033f8ee1d3241e32bf36228318a61f1ace579df4e8170d"
"sha256:0cfe07133fd00b27edee5e6385e333e9eeb010607e8a46e1cd673f05f8596595",
"sha256:11a1f3816ea82eed4178102c56281782690ab5993251fdfd75039aad4d20385f",
"sha256:2762331de395739c91f1abb88041f94a080cb1143aeec791b3b223976228af3f",
"sha256:283d9de87c0133ef98f93dfc09fad3fb382f2a15580de75c02b5bb36a5a159a5",
"sha256:3d22662b4b10112c545c91a0741f2436f8ca979ab3d69d03d19322aa970f9695",
"sha256:41388e32e40b41dd56eb37fcaa7488b2b47b0adf77c66154d6b89622c110dfe9",
"sha256:42c16cec1c8cf2728f1d539bd55aaa9d6bb48a7de2f41eb944697293ef65a559",
"sha256:47ee7a839f5885bc0c63a74aabb91f6f40d7d7b639253768c4199b37aede7982",
"sha256:5a311ee4d983c487a0ab546708edbdd759393a3dc9cd30305170149fedd23c88",
"sha256:5dc65644f75a4c2970f21394ad8bea1a844104f0fe01f278631be1c7eae27226",
"sha256:6ed0d073a9c54ac40c41a9c2d53fcc3d4d4ed607670b9e7b0de1ba13b4cbfe6f",
"sha256:76ba7c40e80f9dc815c5e896330700fd6e20814e69da9c1267d65a4d051080f1",
"sha256:818b9be7900e8dc23e013a92779135623476f44a0de58b40c32a15368c01d471",
"sha256:a024181d7aef0004d76fb3bce2a4c9f2e67a609a9e2a6ff2571d30e9976aa383",
"sha256:a955e4128ac36797aaffd49ab44ec74a71c11d6938df83b1285492d277db5397",
"sha256:a97a954a8c2f046d3817c2bce16e3c7e9a9c2afffaf0400f5c16df5172a67c9c",
"sha256:a97e82c39d9856fe7d4f9b86d8a1e66eff99cf3a8b7ba48202f659703d27c46f",
"sha256:b55b953a1bdb465f4dc181758570d321db4ac23005f90ffd2b434cc6609a63dd",
"sha256:bb02929b0d6bfab4c48a79bd805bd7419114606947ec8284476167415171f55b",
"sha256:bece0a4a49e60e472a6d1f70ac6cdea00f9ab80ff01132f96bd970cdd8a9e5a9",
"sha256:e41e8951749c4b5c9a2dc5fdbc1a4eec6ab2a140fdae9b460b0f557eed870f4d",
"sha256:f71d57cc8645f14816ae249407d309be250ad8de93ef61d9709b45a0ddf4050c"
],
"markers": "python_version < '3.10' and platform_machine != 'aarch64' and platform_machine != 'arm64'",
"version": "==1.21.4"
"version": "==1.22.0"
},
"pandas": {
"hashes": [
"sha256:003ba92db58b71a5f8add604a17a059f3068ef4e8c0c365b088468d0d64935fd",
"sha256:10e10a2527db79af6e830c3d5842a4d60383b162885270f8cffc15abca4ba4a9",
"sha256:22808afb8f96e2269dcc5b846decacb2f526dd0b47baebc63d913bf847317c8f",
"sha256:2d1dc09c0013d8faa7474574d61b575f9af6257ab95c93dcf33a14fd8d2c1bab",
"sha256:35c77609acd2e4d517da41bae0c11c70d31c87aae8dd1aabd2670906c6d2c143",
"sha256:372d72a3d8a5f2dbaf566a5fa5fa7f230842ac80f29a931fb4b071502cf86b9a",
"sha256:42493f8ae67918bf129869abea8204df899902287a7f5eaf596c8e54e0ac7ff4",
"sha256:4acc28364863127bca1029fb72228e6f473bb50c32e77155e80b410e2068eeac",
"sha256:5298a733e5bfbb761181fd4672c36d0c627320eb999c59c65156c6a90c7e1b4f",
"sha256:5ba0aac1397e1d7b654fccf263a4798a9e84ef749866060d19e577e927d66e1b",
"sha256:9707bdc1ea9639c886b4d3be6e2a45812c1ac0c2080f94c31b71c9fa35556f9b",
"sha256:a2aa18d3f0b7d538e21932f637fbfe8518d085238b429e4790a35e1e44a96ffc",
"sha256:a388960f979665b447f0847626e40f99af8cf191bce9dc571d716433130cb3a7",
"sha256:a51528192755f7429c5bcc9e80832c517340317c861318fea9cea081b57c9afd",
"sha256:b528e126c13816a4374e56b7b18bfe91f7a7f6576d1aadba5dee6a87a7f479ae",
"sha256:c1aa4de4919358c5ef119f6377bc5964b3a7023c23e845d9db7d9016fa0c5b1c",
"sha256:c2646458e1dce44df9f71a01dc65f7e8fa4307f29e5c0f2f92c97f47a5bf22f5",
"sha256:c2f44425594ae85e119459bb5abb0748d76ef01d9c08583a667e3339e134218e",
"sha256:d47750cf07dee6b55d8423471be70d627314277976ff2edd1381f02d52dbadf9",
"sha256:d99d2350adb7b6c3f7f8f0e5dfb7d34ff8dd4bc0a53e62c445b7e43e163fce63",
"sha256:dd324f8ee05925ee85de0ea3f0d66e1362e8c80799eb4eb04927d32335a3e44a",
"sha256:eaca36a80acaacb8183930e2e5ad7f71539a66805d6204ea88736570b2876a7b",
"sha256:f567e972dce3bbc3a8076e0b675273b4a9e8576ac629149cf8286ee13c259ae5",
"sha256:fe48e4925455c964db914b958f6e7032d285848b7538a5e1b19aeb26ffaea3ec"
"sha256:1e4285f5de1012de20ca46b188ccf33521bff61ba5c5ebd78b4fb28e5416a9f1",
"sha256:2651d75b9a167cc8cc572cf787ab512d16e316ae00ba81874b560586fa1325e0",
"sha256:2c21778a688d3712d35710501f8001cdbf96eb70a7c587a3d5613573299fdca6",
"sha256:32e1a26d5ade11b547721a72f9bfc4bd113396947606e00d5b4a5b79b3dcb006",
"sha256:3345343206546545bc26a05b4602b6a24385b5ec7c75cb6059599e3d56831da2",
"sha256:344295811e67f8200de2390093aeb3c8309f5648951b684d8db7eee7d1c81fb7",
"sha256:37f06b59e5bc05711a518aa10beaec10942188dccb48918bb5ae602ccbc9f1a0",
"sha256:552020bf83b7f9033b57cbae65589c01e7ef1544416122da0c79140c93288f56",
"sha256:5cce0c6bbeb266b0e39e35176ee615ce3585233092f685b6a82362523e59e5b4",
"sha256:5f261553a1e9c65b7a310302b9dbac31cf0049a51695c14ebe04e4bfd4a96f02",
"sha256:60a8c055d58873ad81cae290d974d13dd479b82cbb975c3e1fa2cf1920715296",
"sha256:62d5b5ce965bae78f12c1c0df0d387899dd4211ec0bdc52822373f13a3a022b9",
"sha256:7d28a3c65463fd0d0ba8bbb7696b23073efee0510783340a44b08f5e96ffce0c",
"sha256:8025750767e138320b15ca16d70d5cdc1886e8f9cc56652d89735c016cd8aea6",
"sha256:8b6dbec5f3e6d5dc80dcfee250e0a2a652b3f28663492f7dab9a24416a48ac39",
"sha256:a395692046fd8ce1edb4c6295c35184ae0c2bbe787ecbe384251da609e27edcb",
"sha256:a62949c626dd0ef7de11de34b44c6475db76995c2064e2d99c6498c3dba7fe58",
"sha256:aaf183a615ad790801fa3cf2fa450e5b6d23a54684fe386f7e3208f8b9bfbef6",
"sha256:adfeb11be2d54f275142c8ba9bf67acee771b7186a5745249c7d5a06c670136b",
"sha256:b6b87b2fb39e6383ca28e2829cddef1d9fc9e27e55ad91ca9c435572cdba51bf",
"sha256:bd971a3f08b745a75a86c00b97f3007c2ea175951286cdda6abe543e687e5f2f",
"sha256:c69406a2808ba6cf580c2255bcf260b3f214d2664a3a4197d0e640f573b46fd3",
"sha256:d3bc49af96cd6285030a64779de5b3688633a07eb75c124b0747134a63f4c05f",
"sha256:fd541ab09e1f80a2a1760032d665f6e032d8e44055d602d65eeea6e6e85498cb",
"sha256:fe95bae4e2d579812865db2212bb733144e34d0c6785c0685329e5b60fcb85dd"
],
"index": "pypi",
"version": "==1.3.4"
"version": "==1.3.5"
},
"pyarrow": {
"hashes": [
......@@ -219,11 +213,11 @@
},
"pygments": {
"hashes": [
"sha256:b8e67fe6af78f492b3c4b3e2970c0624cbf08beb1e493b2c99b9fa1b67a20380",
"sha256:f398865f7eb6874156579fdf36bc840a03cab64d1cde9e93d68f46a425ec52c6"
"sha256:44238f1b60a76d78fc8ca0528ee429702aae011c265fe6a8dd8b63049ae41c65",
"sha256:4e426f72023d88d03b2fa258de560726ce890ff3b630f88c21cbb8b2503b8c6a"
],
"markers": "python_full_version >= '3.5.0'",
"version": "==2.10.0"
"version": "==2.11.2"
},
"python-dateutil": {
"hashes": [
......@@ -316,11 +310,11 @@
"develop": {
"anyio": {
"hashes": [
"sha256:24adc69309fb5779bc1e06158e143e0b6d2c56b302a3ac3de3083c705a6ed39d",
"sha256:2855a9423524abcdd652d942f8932fda1735210f77a6b392eafd9ff34d3fe020"
"sha256:a0aeffe2fb1fdf374a8e4b471444f0f3ac4fb9f5a5b542b48824475e0042a5a6",
"sha256:b5fa16c5ff93fa1046f2eeb5bbff2dad4d3514d6cda61d02816dba34fa8c3c2e"
],
"markers": "python_full_version >= '3.6.2'",
"version": "==3.4.0"
"version": "==3.5.0"
},
"atomicwrites": {
"hashes": [
......@@ -332,11 +326,11 @@
},
"attrs": {
"hashes": [
"sha256:149e90d6d8ac20db7a955ad60cf0e6881a3f20d37096140088356da6c716b0b1",
"sha256:ef6aaac3ca6cd92904cdd0d83f629a15f18053ec84e6432106f7a4d04ae4f5fb"
"sha256:2d27e3784d7a565d36ab851fe94887c5eccd6a463168875832a1be79c82828b4",
"sha256:626ba8234211db98e869df76230a137c4c40a12d72445c45d5f5b716f076e2fd"
],
"markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
"version": "==21.2.0"
"version": "==21.4.0"
},
"certifi": {
"hashes": [
......@@ -347,11 +341,11 @@
},
"charset-normalizer": {
"hashes": [
"sha256:735e240d9a8506778cd7a453d97e817e536bb1fc29f4f6961ce297b9c7a917b0",
"sha256:83fcdeb225499d6344c8f7f34684c2981270beacc32ede2e669e94f7fa544405"
"sha256:876d180e9d7432c5d1dfd4c5d26b72f099d503e8fcc0feb7532c9289be60fcbd",
"sha256:cb957888737fc0bbcd78e3df769addb41fd1ff8cf950dc9e7ad7793f1bf44455"
],
"markers": "python_full_version >= '3.5.0'",
"version": "==2.0.8"
"version": "==2.0.10"
},
"colorama": {
"hashes": [
......@@ -379,19 +373,19 @@
},
"httpcore": {
"hashes": [
"sha256:9a98d2416b78976fc5396ff1f6b26ae9885efbb3105d24eed490f20ab4c95ec1",
"sha256:d10162a63265a0228d5807964bd964478cbdb5178f9a2eedfebb2faba27eef5d"
"sha256:9410fe352bea732311f2b2bee0555c8cc5e62b9a73b9d3272fe125a2aa6eb28e",
"sha256:d4305811f604d3c2e22869147392f134796976ff946c96a8cfba87f4e0171d83"
],
"markers": "python_version >= '3.6'",
"version": "==0.14.3"
"version": "==0.14.4"
},
"httpx": {
"hashes": [
"sha256:02af20df486b78892a614a7ccd4e4e86a5409ec4981ab0e422c579a887acad83",
"sha256:208e5ef2ad4d105213463cfd541898ed9d11851b346473539a8425e644bb7c66"
"sha256:7a3eb67ef0b8abbd6d9402248ef2f84a76080fa1c839f8662e6eb385640e445a",
"sha256:df9a0fd43fa79dbab411d83eb1ea6f7a525c96ad92e60c2d7f40388971b25777"
],
"index": "pypi",
"version": "==0.21.1"
"version": "==0.21.3"
},
"idna": {
"hashes": [
......@@ -472,11 +466,11 @@
},
"respx": {
"hashes": [
"sha256:1ac1cc99bf892ffd3e33108ae43d71d8309a58ac226965f4bd81ec055600f265",
"sha256:4a09e15803c7450d45303520ec528794c9fd77b05984263bc83b78aabbb39413"
"sha256:4b13ba2aa4fc619ad5523786bacb4a565fab4ba3d02582b12f23942aa5a9bdf6",
"sha256:df26cf743f4c48bb38f134a3b0ec7384d5350671264c1c28bbddf41658d6f01a"
],
"index": "pypi",
"version": "==0.19.0"
"version": "==0.19.1"
},
"rfc3986": {
"extras": [
......
......@@ -12,5 +12,54 @@
"viewers": ["data.default.viewers@opendes.contoso.com"],
"owners": ["data.default.owners@opendes.contoso.com"]
}
},
"wellbore_mapping": {
"kind": "osdu:wks:master-data--Wellbore:1.0.0",
"mapping": {
"acl.viewers": "CONFIGURATION.data.default.viewers",
"acl.owners": "CONFIGURATION.data.default.owners",
"legal.legaltags": "CONFIGURATION.legal.legaltags",
"legal.otherRelevantDataCountries": "CONFIGURATION.legal.otherRelevantDataCountries",
"legal.status": "CONFIGURATION.legal.status",
"data.FacilityName": "well.WELL.value",
"data.NameAliases": {
"type": "function",
"function": "build_wellbore_name_aliases",
"args": ["well.UWI.value", "CONFIGURATION.data_partition_id"]
}
}
},
"welllog_mapping": {
"kind": "osdu:wks:work-product-component--WellLog:1.1.0",
"mapping":
{
"acl.viewers": "CONFIGURATION.data.default.viewers",
"acl.owners": "CONFIGURATION.data.default.owners",
"legal.legaltags": "CONFIGURATION.legal.legaltags",
"legal.otherRelevantDataCountries": "CONFIGURATION.legal.otherRelevantDataCountries",
"legal.status": "CONFIGURATION.legal.status",
"data.ReferenceCurveID": "curves[0].mnemonic",
"data.WellboreID": {
"type": "function",
"function": "get_wellbore_id",
"args": []
},
"data.Curves": {
"type": "array",
"source": "curves",
"mapping": {
"CurveID": "mnemonic",
"Mnemonic": "mnemonic",
"CurveUnit": {
"type": "function",
"function": "las2osdu_curve_uom_converter",
"args": [
"unit",
"CONFIGURATION.data_partition_id"
]
}
}
}
}
}
}
\ No newline at end of file
}
......@@ -12,6 +12,7 @@ IFileLoader = importlib.import_module(f"{TARGET_MODULE_NAME}.file_loader").IFile
class TestConfiguration:
_config = {
"base_url": "https://osdu-ship.msft-osdu-test.org",
"data_partition_id": "opendes",
"acl_domain": "contoso.com",
"legal":
......@@ -25,7 +26,9 @@ class TestConfiguration:
"viewers": ["data.default.viewers@opendes.contoso.com"],
"owners": ["data.default.owners@opendes.contoso.com"]
}
}
},
"wellbore_mapping": {"kind": "osdu:wks:master-data--Wellbore:1.0.0"},
"welllog_mapping": {"kind": "osdu:wks:work-product-component--WellLog:1.1.0"}
}
def test_parses_json_as_expected(self):
......@@ -37,11 +40,21 @@ class TestConfiguration:
conf = Configuration(mock_reader, "some_filename.json")
# Assert
assert conf.acl_domain == self._config["acl_domain"]
assert conf.base_url == self._config["base_url"]
assert conf.data_partition_id == self._config["data_partition_id"]
assert conf.legal_relevant_data_countries == self._config["legal"]["otherRelevantDataCountries"]
assert conf.legal_status == self._config["legal"]["status"]
assert conf.legal_tags == self._config["legal"]["legaltags"]
assert conf.data_default_owners == self._config["data"]["default"]["owners"]
assert conf.data_default_viewers == self._config["data"]["default"]["viewers"]
assert conf.wellbore_mapping == self._config["wellbore_mapping"]
assert conf.welllog_mapping == self._config["welllog_mapping"]
def test_get_recursive_returns_expected_result(self):
# Assemble
mock_reader = Mock(spec=IFileLoader)
mock_reader.load.return_value = json.dumps(self._config)
# Act
conf = Configuration(mock_reader, "some_filename.json")
# Assert
assert conf.get_recursive(["acl_domain"]) == self._config["acl_domain"]
assert conf.get_recursive(["legal", "legaltags"]) == self._config["legal"]["legaltags"]
assert conf.get_recursive(["data", "default", "owners"]) == self._config["data"]["default"]["owners"]
......@@ -70,7 +70,7 @@ class TestOsduClient:
# Assemble
base_url = "http://test.bp.com"
record = Record("kind", {}, "legal", {})
record = Record({"kind": "kind", "acl": {}, "legal": "legal", "data": {}})
respx_mock.post(f"{base_url}/api/os-wellbore-ddms/ddms/v3/wellbores").mock(side_effect=self.match_and_mock_post)
......@@ -86,7 +86,7 @@ class TestOsduClient:
# Assemble
base_url = "http://test.bp.com"
record = Record("kind", {}, "legal", {})
record = Record({"kind": "kind", "acl": {}, "legal": "legal", "data": {}})
mock_response = httpx.Response(400)
respx_mock.post(f"{base_url}/api/os-wellbore-ddms/ddms/v3/wellbores").mock(return_value=mock_response)
......@@ -105,7 +105,7 @@ class TestOsduClient:
# Assemble
base_url = "http://test.bp.com"
record = Record("kind", {}, "legal", {})
record = Record({"kind": "kind", "acl": {}, "legal": "legal", "data": {}})
respx_mock.post(f"{base_url}/api/os-wellbore-ddms/ddms/v3/welllogs").mock(side_effect=self.match_and_mock_post)
......@@ -121,7 +121,7 @@ class TestOsduClient:
# Assemble
base_url = "http://test.bp.com"
record = Record("kind", {}, "legal", {})
record = Record({"kind": "kind", "acl": {}, "legal": "legal", "data": {}})
mock_response = httpx.Response(500)
respx_mock.post(f"{base_url}/api/os-wellbore-ddms/ddms/v3/welllogs").mock(return_value=mock_response)
......
import pytest
from importlib import import_module
from unittest.mock import Mock, call
from . import TARGET_MODULE_NAME
# Test target module import
property_mapping = import_module(f"{TARGET_MODULE_NAME}.property_mapping")
PropertyMapper = property_mapping.PropertyMapper
IPropertyMappingLoader = property_mapping.IPropertyMappingLoader
IDictionaryLoader = import_module(f"{TARGET_MODULE_NAME}.file_loader").IDictionaryLoader
WdutilValidationException = import_module(f"{TARGET_MODULE_NAME}.exceptions").WbdutilValidationException
WbdutilAttributeError = import_module(f"{TARGET_MODULE_NAME}.exceptions").WbdutilAttributeError
Configuration = import_module(f"{TARGET_MODULE_NAME}.configuration").Configuration
IFileLoader = import_module(f"{TARGET_MODULE_NAME}.file_loader").IFileLoader
class TestPropertyMapper:
def test_GIVEN_flat_mapping_WHEN_remap_data_THEN_returns_expected_object(self):
# Arrange
mapping = {
"xyz": "abc",
"uvw": "ef"
}
mock_reader = Mock(spec=IPropertyMappingLoader)
mock_reader.mapping = mapping
target = PropertyMapper(mock_reader, None, None)
source = {
"abc": "some data",
"ef": ["An", "Array"]
}
# Act
result = target.remap_data(source)
# Assert
assert result["xyz"] is source["abc"]
assert result["uvw"] is source["ef"]
def test_GIVEN_nested_source_mapping_WHEN_remap_data_THEN_returns_expected_object(self):
# Arrange
mapping = {
"dest1": "level1a.level2.level3",
"dest2": "level1b.level2",
"dest3": "level1a"
}
mock_reader = Mock(spec=IPropertyMappingLoader)
mock_reader.mapping = mapping
target = PropertyMapper(mock_reader, None, None)
source = {
"level1a": {
"level2": {
"level3": "a string"
}
},
"level1b": {
"level2":