Wellbore Domain Services merge requestshttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests2023-08-17T11:23:51Zhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/158Fix duplicates in openapi generation, unit test to detect duplicates included2023-08-17T11:23:51Zfabian serinFix duplicates in openapi generation, unit test to detect duplicates includedFix duplicates in openapi generation, unit test to detect duplicates includedFix duplicates in openapi generation, unit test to detect duplicates includedM7 - Release 0.10https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/155Add a fast check on columns type from input data to return a 422 and explicit...2023-08-17T11:23:54ZCyril MonmoutonAdd a fast check on columns type from input data to return a 422 and explicit...Add a fast check on columns type from input data to return a 422 and explicit error message, instead of ugly internal errorAdd a fast check on columns type from input data to return a 422 and explicit error message, instead of ugly internal errorM7 - Release 0.10YannickVincent RondotJeremie HallalYannickhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/153Add v3 bulk tracing2023-08-17T11:23:57ZCyril MonmoutonAdd v3 bulk tracingM7 - Release 0.10https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/152Bulk api v2 vd2023-08-17T11:23:58ZVictor DosdaBulk api v2 vdM7 - Release 0.10Victor DosdaVictor Dosdahttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/150Fusion basic integration CRUD tests, with recordIdVersions check on local FS2023-08-17T11:24:00ZLuc YriarteFusion basic integration CRUD tests, with recordIdVersions check on local FSFusion basic integration CRUD tests, add recordIdVersions check to test_crud_create_record and remove test_crud_create_record_check_versions
Add missing recordIdVersions list to the CreateUpdateRecordsResponse in the local FS storage imp...Fusion basic integration CRUD tests, add recordIdVersions check to test_crud_create_record and remove test_crud_create_record_check_versions
Add missing recordIdVersions list to the CreateUpdateRecordsResponse in the local FS storage implementationM7 - Release 0.10Luc YriarteLuc Yriartehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/148[bug] fix bug on string curves2023-08-17T11:24:03ZJeremie Hallal[bug] fix bug on string curvesUpdate a WellLog with a column of type string starting with some nan values
ex:
|A | strC |
|:---:|:------:|
| 1 | nan |
| 2 | nan |
| 3 | abc |
Exception: ArrowInvalid('Could not convert string_value_0 with type str: tried to...Update a WellLog with a column of type string starting with some nan values
ex:
|A | strC |
|:---:|:------:|
| 1 | nan |
| 2 | nan |
| 3 | abc |
Exception: ArrowInvalid('Could not convert string_value_0 with type str: tried to convert to double', 'Conversion failed for column strC with type object')M7 - Release 0.10https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/147changed dask version in requiremet.txt2023-08-17T11:24:05ZParomita Mitrachanged dask version in requiremet.txtChanged dask[distributed] version as service was failing to start in QA
https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/pipelines/49238Changed dask[distributed] version as service was failing to start in QA
https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/pipelines/49238M7 - Release 0.10Anuj GuptaAnuj Guptahttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/146Update fossa notice2023-08-17T11:24:06ZLuc YriarteUpdate fossa noticeM7 - Release 0.10https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/145local fs storage enhancements2023-08-17T11:24:08ZJeremie Hallallocal fs storage enhancementsM7 - Release 0.10https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/144Properly raise bulk not found exception on get data version with no bulk urn2023-08-17T11:24:09ZLuc YriarteProperly raise bulk not found exception on get data version with no bulk urnThe unit test happened to be wrong, did not catch the issue before.The unit test happened to be wrong, did not catch the issue before.M7 - Release 0.10Luc YriarteLuc Yriartehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/143add tests to ensure multiple sessions in parallel2023-08-17T11:24:11ZJeremie Hallaladd tests to ensure multiple sessions in parallelM7 - Release 0.10https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/142Fix wrong raise in app_injector.make2023-08-17T11:24:12ZChristophe LallementFix wrong raise in app_injector.makefix app_injector.make raise object NotImplemented() instead NotImplementedError() exceptionfix app_injector.make raise object NotImplemented() instead NotImplementedError() exceptionM7 - Release 0.10Christophe LallementChristophe Lallementhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/141Implement columns selection patterns2023-08-17T11:24:14ZJeremie HallalImplement columns selection patternsM7 - Release 0.10https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/140WellLog 100 schema update and forbid extra fields2023-08-17T11:24:15Zfabian serinWellLog 100 schema update and forbid extra fieldsWellLog 100 schema update and forbid extra fieldsWellLog 100 schema update and forbid extra fieldsM7 - Release 0.10https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/139fix some bottleneck found during perf test failure investigation2023-08-17T11:24:17ZYannickfix some bottleneck found during perf test failure investigationfix some bottleneck found during perf test failure investigation. Fixes are:
On Azure lib, not in this MR [see corresponding MR](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/lib/wellbore-cloud/wellbor...fix some bottleneck found during perf test failure investigation. Fixes are:
On Azure lib, not in this MR [see corresponding MR](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/lib/wellbore-cloud/wellbore-azure-lib/-/merge_requests/22)
- Secret value pinned to partition info (cached from 5 minutes), _reason_ = get secret is done synchronous for each request causing the service to be unresponsive in average during 90ms.
- Add an asyncio lock after unsuccessfully checking cache, _reason_ = prevent unnecessary call to fetch partition info and lead to cache reset in case on multiple concurrency.
**WDMS Service, this MR**
- Extend backoff to exception client_storage.ResponseHandlingException, _reason_ = Error such as "server disconnected" are caught and rethrown as ResponseHandlingException. Is that case provide some resilience, which tends to occur during stress.
- Bulk data (de)serialization [done asynchronously](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/blob/fix_perf/app/bulk_persistence/dataframe_serializer.py#L110). _reason_ = this appears to be the main bottleneck during perf test. Since the task is done synchronously, (de)serialization of bulk may take few 10 ms to few seconds to complete. The service becomes fully unresponsive during this time. The fix is to execute these tasks inside process pool (in favor of a thread pool since it's a CPU bound operation) rather than synchronously inside the main thread. For now the pool size is arbitrary set to 4 (subject to change later).
There's also some [bootstrap mechanism](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/blob/fix_perf/app/wdms_app.py#L123) at service startup, so pool is fully ready at first incoming request.M7 - Release 0.10https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/138content type header addition2023-08-17T11:24:19ZParomita Mitracontent type header additionM7 - Release 0.10Anuj GuptaParomita MitraAnuj Guptahttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/137Adding search API for osdu objects2023-08-17T11:24:20Zfabian serinAdding search API for osdu objectsAdding search API for osdu objectsAdding search API for osdu objectsM7 - Release 0.10https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/136Bulk api v22023-08-17T11:24:22ZLuc YriarteBulk api v2M7 - Release 0.10https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/134Refactor record access utilities from logs/v22023-08-17T11:24:25ZLuc YriarteRefactor record access utilities from logs/v2M7 - Release 0.10Luc YriarteLuc Yriartehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/133trajectory bulk V3 (common bulk v3 router)2023-08-17T11:24:26ZYannicktrajectory bulk V3 (common bulk v3 router)trajectory bulk V3 (common bulk v3 router)trajectory bulk V3 (common bulk v3 router)M7 - Release 0.10