Wellbore Domain Services merge requestshttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests2023-08-17T11:24:44Zhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/121Fix wrong status code in Tracing requests when calling external dependencies2023-08-17T11:24:44ZCyril MonmoutonFix wrong status code in Tracing requests when calling external dependenciesAs it's done in "app/middleware/traces_middleware.py" surround call to ensure 500 error code is set in case of raised Exceptions.As it's done in "app/middleware/traces_middleware.py" surround call to ensure 500 error code is set in case of raised Exceptions.M7 - Release 0.10YannickChristophe LallementYannickhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/122fix dipSet version in kind2023-08-17T11:24:42ZVictor Dosdafix dipSet version in kindM7 - Release 0.10Victor DosdaVictor Dosdahttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/123Remove dask folder2023-08-17T11:24:41ZJeremie HallalRemove dask folderremove the dask_data folder from the blob root path.
add and init a logger into dask workers processremove the dask_data folder from the blob root path.
add and init a logger into dask workers processM7 - Release 0.10https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/124Update fastapi to update pydantic to remove CVE-2021-29510 from vulnerabilities2023-08-17T11:24:39ZKin Jin NgUpdate fastapi to update pydantic to remove CVE-2021-29510 from vulnerabilities- Update fastapi to update pydantic to address CVE-2021-29510 in vulnerability scan
- Update frozen requirements with pip freeze- Update fastapi to update pydantic to address CVE-2021-29510 in vulnerability scan
- Update frozen requirements with pip freezeM7 - Release 0.10https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/125Enable multiple post build templates2023-08-17T11:24:37ZKin Jin NgEnable multiple post build templates- Allow multiple post build templates to enable further customization of pipeline- Allow multiple post build templates to enable further customization of pipelineM7 - Release 0.10https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/126Fix openapi file out of sync2023-08-17T11:24:36ZKin Jin NgFix openapi file out of sync- openapi.json auto generated spec file got out of sync- openapi.json auto generated spec file got out of syncM7 - Release 0.10https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/127Update notice2023-08-17T11:24:34ZKin Jin NgUpdate notice- Update NOTICE generated in build job https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/jobs/385694- Update NOTICE generated in build job https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/jobs/385694M7 - Release 0.10https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/128Update README on par with platform API documentation2023-08-17T11:24:32ZLuc YriarteUpdate README on par with platform API documentationhttps://community.opengroup.org/osdu/documentation/-/tree/master/platform/api/Wellbore-DDMShttps://community.opengroup.org/osdu/documentation/-/tree/master/platform/api/Wellbore-DDMSM7 - Release 0.10Luc YriarteLuc Yriartehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/129Enable module local switch to skip extension2023-08-17T11:24:31ZKin Jin NgEnable module local switch to skip extension- Enable module local switch to skip adding extension routers
- Update doc- Enable module local switch to skip adding extension routers
- Update docM7 - Release 0.10https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/130[Clients middleware] Tracing-WrongErrorCode/Bug: initialize correctly respons...2023-08-17T11:24:29ZCyril Monmouton[Clients middleware] Tracing-WrongErrorCode/Bug: initialize correctly response variable to ensure "if response" can runM7 - Release 0.10YannickVincent RondotKin Jin NgYannickhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/131[WDMS Observability] Add request and response headers: Content-type and Conte...2023-08-17T11:24:28ZCyril Monmouton[WDMS Observability] Add request and response headers: Content-type and Content-length as attributes in traced RequestM7 - Release 0.10Vincent RondotChristophe LallementVincent Rondothttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/132Add node selector2021-09-02T12:20:37ZDzmitry_Paulouski (slb)Add node selectorAll Submissions:
[YES] Have you followed our code review guidelines?
[YES] Have you added an explanation of what your changes do and why you'd like us to include them?
[NO] I have updated the documentation accordingly.
[NA] I have adde...All Submissions:
[YES] Have you followed our code review guidelines?
[YES] Have you added an explanation of what your changes do and why you'd like us to include them?
[NO] I have updated the documentation accordingly.
[NA] I have added tests to cover my changes.
[YES] All new and existing tests passed.
[YES] My code follows the code style of this project.
[YES] I ran lint checks locally prior to submission.
What is the current behavior?
Currently, we have only one node pool for services and system workloads.
What is the new behavior?
We are going to move workloads to a new node pool with enabled autoscaling. Here you can find the details:
Issue link
Does this introduce a breaking change?
[NO]
This change has a strong dependency on the following MR and should be merged only after it:
osdu/platform/deployment-and-operations/infra-azure-provisioning!324https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/133trajectory bulk V3 (common bulk v3 router)2023-08-17T11:24:26ZYannicktrajectory bulk V3 (common bulk v3 router)trajectory bulk V3 (common bulk v3 router)trajectory bulk V3 (common bulk v3 router)M7 - Release 0.10https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/134Refactor record access utilities from logs/v22023-08-17T11:24:25ZLuc YriarteRefactor record access utilities from logs/v2M7 - Release 0.10Luc YriarteLuc Yriartehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/135V3 search on OSDU entities2021-06-22T09:12:32Zfabian serinV3 search on OSDU entitiesAdd V3 api for searching OSDU entitiesAdd V3 api for searching OSDU entitieshttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/136Bulk api v22023-08-17T11:24:22ZLuc YriarteBulk api v2M7 - Release 0.10https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/137Adding search API for osdu objects2023-08-17T11:24:20Zfabian serinAdding search API for osdu objectsAdding search API for osdu objectsAdding search API for osdu objectsM7 - Release 0.10https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/138content type header addition2023-08-17T11:24:19ZParomita Mitracontent type header additionM7 - Release 0.10Anuj GuptaParomita MitraAnuj Guptahttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/139fix some bottleneck found during perf test failure investigation2023-08-17T11:24:17ZYannickfix some bottleneck found during perf test failure investigationfix some bottleneck found during perf test failure investigation. Fixes are:
On Azure lib, not in this MR [see corresponding MR](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/lib/wellbore-cloud/wellbor...fix some bottleneck found during perf test failure investigation. Fixes are:
On Azure lib, not in this MR [see corresponding MR](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/lib/wellbore-cloud/wellbore-azure-lib/-/merge_requests/22)
- Secret value pinned to partition info (cached from 5 minutes), _reason_ = get secret is done synchronous for each request causing the service to be unresponsive in average during 90ms.
- Add an asyncio lock after unsuccessfully checking cache, _reason_ = prevent unnecessary call to fetch partition info and lead to cache reset in case on multiple concurrency.
**WDMS Service, this MR**
- Extend backoff to exception client_storage.ResponseHandlingException, _reason_ = Error such as "server disconnected" are caught and rethrown as ResponseHandlingException. Is that case provide some resilience, which tends to occur during stress.
- Bulk data (de)serialization [done asynchronously](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/blob/fix_perf/app/bulk_persistence/dataframe_serializer.py#L110). _reason_ = this appears to be the main bottleneck during perf test. Since the task is done synchronously, (de)serialization of bulk may take few 10 ms to few seconds to complete. The service becomes fully unresponsive during this time. The fix is to execute these tasks inside process pool (in favor of a thread pool since it's a CPU bound operation) rather than synchronously inside the main thread. For now the pool size is arbitrary set to 4 (subject to change later).
There's also some [bootstrap mechanism](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/blob/fix_perf/app/wdms_app.py#L123) at service startup, so pool is fully ready at first incoming request.M7 - Release 0.10https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/merge_requests/140WellLog 100 schema update and forbid extra fields2023-08-17T11:24:15Zfabian serinWellLog 100 schema update and forbid extra fieldsWellLog 100 schema update and forbid extra fieldsWellLog 100 schema update and forbid extra fieldsM7 - Release 0.10