Storage merge requestshttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests2022-12-22T18:20:13Zhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/536DO NOT MERGE2022-12-22T18:20:13Zshivani karipeDO NOT MERGE# Merge request template# Merge request templatehttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/439DRAFT: add souts2022-12-22T18:27:03ZRostislav Vatolinvatolinrp@gmail.comDRAFT: add souts# Merge request template# Merge request templateRostislav Vatolinvatolinrp@gmail.comRostislav Vatolinvatolinrp@gmail.comhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/590Collaboration Feature Flag2023-01-17T18:19:06ZAnfal FadhilCollaboration Feature FlagAdded a partition specific feature flag for collaboration.Added a partition specific feature flag for collaboration.Anfal FadhilAnfal Fadhilhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/608message properties updated.2023-02-02T15:46:49ZAnfal Fadhilmessage properties updated.A small docs update MR. adding the new properties to the message.A small docs update MR. adding the new properties to the message.Anfal FadhilAnfal Fadhilhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/460MSCOSDU-127 update deployment2023-02-04T05:34:15ZIgor Zimovets (EPAM)MSCOSDU-127 update deploymentChange motivation:
https://github.com/Azure/application-gateway-kubernetes-ingress/blob/master/docs/how-tos/minimize-downtime-during-deployments.mdChange motivation:
https://github.com/Azure/application-gateway-kubernetes-ingress/blob/master/docs/how-tos/minimize-downtime-during-deployments.mdhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/489Updating POM of Storage Service to take up latest code from azure-core-lib2023-02-04T05:34:20ZSrishti SharmaUpdating POM of Storage Service to take up latest code from azure-core-lib# Merge request template
Issue: Stack trace is split across multiple line. Target: Reduce stack trace by having each exception without line breaks. Too many logs. Difficult to debug for DRIs.
Bug: https://dev.azure.com/OpenEnergyPlatfor...# Merge request template
Issue: Stack trace is split across multiple line. Target: Reduce stack trace by having each exception without line breaks. Too many logs. Difficult to debug for DRIs.
Bug: https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/\_workitems/edit/6422
1. Updated POM.xml of Storage Service to take up latest code version(0.17.0-rc6)from azure-core-lib i.e. <osdu.corelibazure.version>0.17.0-rc6</osdu.corelibazure.version>
2. Logging library is already updated with version 0.17.0-rc6 which removes line breaks in stack trace.
Target: Each exception in one line. Reduces log size.https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/424Update Core-Lib-Azure Library Version2023-02-04T05:34:21ZAnubhav AronUpdate Core-Lib-Azure Library VersionOverview: Service use core-lib-azure as external dependency. Updating core-lib-azure version to add feature of adding logging frameworkOverview: Service use core-lib-azure as external dependency. Updating core-lib-azure version to add feature of adding logging frameworkAnubhav AronAnubhav Aronhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/488fix api test2023-02-04T05:34:22ZNikhil Singh[MicroSoft]fix api test# Merge request template# Merge request templateNikhil Singh[MicroSoft]Nikhil Singh[MicroSoft]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/399Updated storage service to use RedisAzureCache2023-02-04T05:34:22ZShiv SinghUpdated storage service to use RedisAzureCache# Merge request template# Merge request templateShiv SinghShiv Singhhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/598Handling data inconsistency properly (GONRG-6319)2023-02-06T16:09:54ZKanstantsin Dubrouski [EPAM / GCP]Handling data inconsistency properly (GONRG-6319)Handling data inconsistency properlyHandling data inconsistency properlyM16 - Release 0.19Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/509Swagger Sanity Phase 2: Using springdoc-openapi for swaggers APIs2023-02-09T10:03:08ZDmitrii Novikov (EPAM)Swagger Sanity Phase 2: Using springdoc-openapi for swaggers APIs## Type of change
- [ ] Bug Fix
- [x] Feature
https://community.opengroup.org/osdu/platform/system/home/-/issues/97
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider imple...## Type of change
- [ ] Bug Fix
- [x] Feature
https://community.opengroup.org/osdu/platform/system/home/-/issues/97
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Using springfox library for swagger
## What is the new/expected behavior?
Using springdoc library for swagger
## Have you added/updated Unit Tests and Integration Tests?
Integration test for swagger endpoint addedSrinivasan NarayananSrinivasan Narayananhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/615Security Fixes2023-02-10T17:48:25ZMarc Burnie [AWS]Security FixesIncludes: AWS Adding support for EKS 1.23Includes: AWS Adding support for EKS 1.23M16 - Release 0.19Marc Burnie [AWS]Marc Burnie [AWS]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/610Add springdoc openapi latest version2023-02-17T14:29:26ZThulasi Dass SubramanianAdd springdoc openapi latest version## Type of change
- [ ] Bug Fix
- [x] Feature
**Link to ADR(Architecture Decision Record)** : [Swagger using springdoc-openapi](https://community.opengroup.org/osdu/platform/system/home/-/issues/97)
## OpenAPI 3.0 related changes
- u...## Type of change
- [ ] Bug Fix
- [x] Feature
**Link to ADR(Architecture Decision Record)** : [Swagger using springdoc-openapi](https://community.opengroup.org/osdu/platform/system/home/-/issues/97)
## OpenAPI 3.0 related changes
- upgraded to latest **springdoc openapi** latest version [1.6.14](https://mvnrepository.com/artifact/org.springdoc/springdoc-openapi-ui/1.6.14)
- used [**@OpenAPIDefinition**](https://docs.swagger.io/swagger-core/v2.0.9/apidocs/io/swagger/v3/oas/annotations/OpenAPIDefinition.html) annotation to populate OpenAPI Object fields _[info, tags, servers, security]_
- Documented **Query, Record, Patch , Info & Health** API with OpenAPI 3.0 **Annotations**
- Added the standard HTTP Response(4xx, 5x****x) for API Responses
- Custom Path for
- **Swagger UI**: https://host/context-path/swagger (will redirect to https://host/context-path/swagger-ui/index.html)
- **api-docs (JSON)** : https://host/context-path/api-docs
- **api-docs (YAML)** : https://host/context-path/api-docs.yaml
## Other Changes
- **Configurable** descriptions managed in [swagger.properties](https://community.opengroup.org/osdu/platform/system/storage/-/blob/az/td-swagger-springdoc-openapi/storage-core/src/main/resources/swagger.properties)
- added **Integration Test** cases for _**Swagger-UI, api-docs**_ endpoints
- Hide **deprecated** [**Schema, PubSubEndpoints**] related API operations in documentation
- Deleted HomeController
## References
- https://springdoc.org/faq.html#_can_i_use_spring_property_with_swagger_annotations
- https://springdoc.org/migrating-from-springfox.htmlThulasi Dass SubramanianThulasi Dass Subramanianhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/612Upgrade First Party Library Dependencies for Release 0.192023-02-18T07:31:17ZDavid Diederichd.diederich@opengroup.orgUpgrade First Party Library Dependencies for Release 0.19This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any...This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any library that is older than the previous release will be left as-is, since the upgrade is likely to be more complicated.
Furthermore, the upgrade should only be merged in the CI pipeline reports success.
If this MR has failed, we can spend a little time investigating to see if a trivial upgrade could achieve compatiblity to the new library.
But significant upgrade efforts should not occur on this MR, as part of the release tagging process.
Instead, significant work should be scheduled for a subsequent milestone.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: 6ccebc445eb48ac1a19bc5fdf305017c14df1dee
Maven: 0.20.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | ---------- | ------------------ |
| core-lib-azure | 0.19.0-rc5 | |
| core-lib-gcp | 0.19.0-rc5 | |
| os-core-lib-aws | 0.17.0 | 0.14.0-rc2 |
| obm | 0.19.0-rc4 | |
| oqm | 0.18.0 | |
| os-core-common | 0.19.0-rc3 | 0.13.0, 0.18.0-rc3 |
| os-core-lib-ibm | 0.17.0-rc4 | 0.17.0-rc4 |
| osm | 0.18.0 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.0 | 2.8.1, 2.13.2.2 |
| (3rd Party) org.springframework.spring-webflux | 5.3.22 | |
| (3rd Party) org.springframework.spring-webmvc | 5.3.22 | 5.3.12, 5.3.22 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade-3
SHA: 1fade6d667e0d7beffe7311810be98e5b6ba4a4c
Maven: 0.20.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | ---------- | ------------------ |
| core-lib-azure | 0.19.0 | |
| core-lib-gcp | 0.19.0 | |
| os-core-lib-aws | 0.17.0 | 0.14.0-rc2 |
| obm | 0.19.0 | |
| oqm | 0.19.0 | |
| os-core-common | 0.19.0 | 0.13.0, 0.18.0-rc3 |
| os-core-lib-ibm | 0.17.0-rc4 | 0.17.0-rc4 |
| osm | 0.19.0 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.0 | 2.8.1, 2.13.2.2 |
| (3rd Party) org.springframework.spring-webflux | 5.3.24 | |
| (3rd Party) org.springframework.spring-webmvc | 5.3.22 | 5.3.12, 5.3.22 |M16 - Release 0.19https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/613Draft: Patch records api (in progress)2023-03-06T14:44:55ZAlok JoshiDraft: Patch records api (in progress)https://community.opengroup.org/osdu/platform/system/storage/-/issues/124https://community.opengroup.org/osdu/platform/system/storage/-/issues/124https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/605Conversion: validate records at the start using json schemas2023-03-10T13:17:20ZMykyta SavchukConversion: validate records at the start using json schemasThere is a bug, when SpatialLocation in record is array, storage returns 500 on batch. Example record data:
```
"SpatialLocation": [
{
"SpatialGeometryTypeID": "Point",
....
]
```
And there were a lot of sim...There is a bug, when SpatialLocation in record is array, storage returns 500 on batch. Example record data:
```
"SpatialLocation": [
{
"SpatialGeometryTypeID": "Point",
....
]
```
And there were a lot of similar bugs related to record geo data validation lately (like misspelling field names and so on, that resulted in 500), but the existing validation logic is not intuitive and is spread across the code with random checks, so to fix them we had to add different checks (if's) in different parts of the code, and to make this logic more robust and prevent future bugs it was decided to do a refactoring and place the validation in one place using json schemas, since we already have them, and remove any further validation checks. Also, as we already have a well defined schema for the SpatialLocation geo attribute, I added a validation for such case in addition to the existing generic geo attribute validation, which I also converted to json schema (GenericGeoAttribute.json).
- Changed conversion logic to add validation against the schema at the start of the flow;
- Removed unnecessary validation checks;
- In case of validation error, the message from validation library is returned, instead of constant predefined messages;
- I changed tests' data (unit and integration), because the schema for AnyCrsFeatureCollection says that if bbox is present, it should not be null, so I added a sample value.M17 - Release 0.20Chad LeongMykyta SavchukChad Leonghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/652Filter metadata records by partition id2023-03-16T20:45:06ZMorris EstepaFilter metadata records by partition id## Type of change
- [X] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
https://community.opengroup.org/osdu/platform/system/storage/-/issues/167
## Does this introduce a change in ...## Type of change
- [X] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
https://community.opengroup.org/osdu/platform/system/storage/-/issues/167
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
metadata records from other data partition ids show up when querying records for a given partition id
## What is the new/expected behavior?
only metadata records from the target partition id shows up in the query results
## Have you added/updated Unit Tests and Integration Tests?
Yes
## Any other useful informationM16 - Release 0.19Morris EstepaMorris Estepahttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/611AWS Merge2023-03-28T15:29:37ZAbhishek PatilAWS MergeSpring Security core version higher than 5.7.5 and other vulnerable package upgradesSpring Security core version higher than 5.7.5 and other vulnerable package upgradesM17 - Release 0.20Okoun-Ola Fabien HouetoYash DholakiaManish JangidOkoun-Ola Fabien Houetohttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/668remove snakeyaml from dependencies2023-04-05T13:08:55ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comremove snakeyaml from dependencies# Merge request template# Merge request templatehttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/676Remove SNAPSHOT dependencies2023-04-13T19:03:21ZDavid Diederichd.diederich@opengroup.orgRemove SNAPSHOT dependenciesThis automated MR removes usage of `SNAPSHOT` versions in the first party library dependencies.
Since `SNAPSHOT` dependencies change frequently -- by their nature -- usage of them across projects is dangerous and should be avoided.
### ...This automated MR removes usage of `SNAPSHOT` versions in the first party library dependencies.
Since `SNAPSHOT` dependencies change frequently -- by their nature -- usage of them across projects is dangerous and should be avoided.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: 181b245edcd31f730b8bfdb855e299c4e413161f
Maven: 0.21.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | --------------- | ---------------------- |
| core-lib-azure | 0.21.0-rc1 | |
| core-lib-gc | 0.21.0-rc1 | |
| os-core-lib-aws | 0.21.0-SNAPSHOT | 0.19.0-rc3 |
| obm | 0.20.0-rc2 | |
| oqm | 0.19.0 | |
| os-core-common | 0.20.0-rc3 | 0.13.0, 0.18.0-rc3 |
| os-core-lib-ibm | 0.17.0-rc4 | 0.17.0-rc4 |
| osm | 0.21.0-rc1 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.0 | 2.8.1, 2.13.2.2 |
| (3rd Party) org.springframework.spring-webmvc | 5.3.26, 5.3.22 | 5.3.12, 5.3.24, 5.3.22 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.27, 1.30 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade-2
SHA: c76288d84a1ce97e663476aea7e68f99d43a4540
Maven: 0.21.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | -------------- | ---------------------- |
| core-lib-azure | 0.21.0-rc1 | |
| core-lib-gc | 0.21.0-rc1 | |
| os-core-lib-aws | 0.20.0 | 0.19.0-rc3 |
| obm | 0.20.0-rc2 | |
| oqm | 0.19.0 | |
| os-core-common | 0.20.0-rc3 | 0.13.0, 0.18.0-rc3 |
| os-core-lib-ibm | 0.17.0-rc4 | 0.17.0-rc4 |
| osm | 0.21.0-rc1 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.0 | 2.8.1, 2.13.2.2 |
| (3rd Party) org.springframework.spring-webmvc | 5.3.26, 5.3.22 | 5.3.12, 5.3.24, 5.3.22 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.27, 1.30 |M17 - Release 0.20