Storage merge requestshttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests2023-06-29T15:34:33Zhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/719Fix instance name2023-06-29T15:34:33ZAleh Shubko [EPAM]Fix instance nameM19 - Release 0.22Madalyn MarabellaMadalyn Marabellahttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/708Merge branch 'gc-containerize-fix' into 'master'2023-06-02T11:13:40ZAliaksandr Ramanovich (EPAM)Merge branch 'gc-containerize-fix' into 'master'# Merge request template
These changes will fix https://community.opengroup.org/osdu/platform/system/storage/-/jobs/1997071 in tagged pipelines# Merge request template
These changes will fix https://community.opengroup.org/osdu/platform/system/storage/-/jobs/1997071 in tagged pipelinesM18 - Release 0.21Aliaksandr Ramanovich (EPAM)Aliaksandr Ramanovich (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/706Upgrade First Party Library Dependencies for Release 0.212023-06-06T05:28:28ZDavid Diederichd.diederich@opengroup.orgUpgrade First Party Library Dependencies for Release 0.21This generated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any...This generated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any library that is older than the previous release will be left as-is, since the upgrade is likely to be more complicated.
Furthermore, the upgrade should only be merged in the CI pipeline reports success.
If this MR has failed, we can spend a little time investigating to see if a trivial upgrade could achieve compatiblity to the new library.
But significant upgrade efforts should not occur on this MR, as part of the release tagging process.
Instead, significant work should be scheduled for a subsequent milestone.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: 2d73acb53394eab3f18c5f95b4ac884577614b14
Maven: 0.22.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | ---------------------- | ---------------------- |
| core-lib-azure | 0.21.0-rc1 | |
| core-lib-gc | 0.21.0-rc4 | |
| os-core-lib-aws | 0.21.0-rc5 | 0.21.0-rc5 |
| obm | 0.21.0-rc2 | |
| oqm | 0.21.0-rc5 | |
| os-core-common | 0.20.0-rc3, 0.21.0-rc4 | 0.13.0, 0.18.0-rc3 |
| os-core-lib-ibm | 0.17.0-rc4 | 0.17.0-rc4 |
| osm | 0.21.0-rc3 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.0 | 2.8.1, 2.13.2.2 |
| (3rd Party) org.springframework.spring-webmvc | 5.3.26, 5.3.22 | 5.3.12, 5.3.24, 5.3.22 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.27, 1.30 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade-2
SHA: 214d0a50d2b6a517b23600aeb6e0f6f447d19488
Maven: 0.22.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | ------------------ | ---------------------- |
| core-lib-azure | 0.21.0 | |
| core-lib-gc | 0.21.0 | |
| os-core-lib-aws | 0.21.0 | 0.21.0 |
| obm | 0.21.0 | |
| oqm | 0.21.0 | |
| os-core-common | 0.20.0-rc3, 0.21.0 | 0.13.0, 0.18.0-rc3 |
| os-core-lib-ibm | 0.17.0-rc4 | 0.17.0-rc4 |
| osm | 0.21.0 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.0 | 2.8.1, 2.13.2.2 |
| (3rd Party) org.springframework.spring-webmvc | 5.3.26, 5.3.22 | 5.3.12, 5.3.24, 5.3.22 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.27, 1.30 |M18 - Release 0.21https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/673Full Upgrade of First Party Library Dependencies for Release 0.202023-05-22T15:18:34ZDavid Diederichd.diederich@opengroup.orgFull Upgrade of First Party Library Dependencies for Release 0.20This generated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to try to fully upgrade all dependent libraries to see if the latest code will work.
It is expected that these will ...This generated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to try to fully upgrade all dependent libraries to see if the latest code will work.
It is expected that these will often fail, since the upgrades were previously rejected for failing pipelines and have not been directly addressed yet.
This upgrade should only be merged in the CI pipeline reports success.
If this MR has failed, we can spend a little time investigating to see if a trivial upgrade could achieve compatiblity to the new library.
But significant upgrade efforts should not occur on this MR, as part of the release tagging process.
Instead, significant work should be scheduled for a subsequent milestone.
This MR may co-exist with a separate, smaller upgrade MR.
If both pass, this one should be used instead.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: f94322cf62a0d949e6a640b209dd2a2fa1ea5ead
Maven: 0.21.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | --------------- | ---------------------- |
| core-lib-azure | 0.20.0-rc8 | |
| core-lib-gc | 0.21.0-rc1 | |
| os-core-lib-aws | 0.21.0-SNAPSHOT | 0.19.0-rc3 |
| obm | 0.20.0-rc2 | |
| oqm | 0.19.0 | |
| os-core-common | 0.20.0-rc3 | 0.13.0, 0.18.0-rc3 |
| os-core-lib-ibm | 0.17.0-rc4 | 0.17.0-rc4 |
| osm | 0.21.0-rc1 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.0 | 2.8.1, 2.13.2.2 |
| (3rd Party) org.springframework.spring-webmvc | 5.3.26, 5.3.22 | 5.3.12, 5.3.24, 5.3.22 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.27, 1.30 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade
SHA: 043bdc96420db671b2ff939b1614d95cd57f3266
Maven: 0.21.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | ---------- | --------------- |
| core-lib-azure | 0.20.0 | |
| core-lib-gc | 0.21.0-rc3 | |
| os-core-lib-aws | 0.20.0 | 0.20.0 |
| obm | 0.20.0 | |
| oqm | 0.20.0 | |
| os-core-common | 0.20.1 | 0.20.1 |
| os-core-lib-ibm | 0.20.0 | 0.20.0 |
| osm | 0.21.0-rc2 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.0 | 2.8.1, 2.13.2.2 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.30, 2.0 |M18 - Release 0.21Okoun-Ola Fabien HouetoBrindaban DasYauhen Shaliou [EPAM/GCP]Srinivasan NarayananOkoun-Ola Fabien Houetohttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/558Upgrading storage service upgrade for the core library dependencies2023-05-13T05:06:02ZNishant VidyasagarUpgrading storage service upgrade for the core library dependencies# Merge request template# Merge request templateNishant VidyasagarNishant Vidyasagarhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/557Upgrading storage service for the core library dependencies2023-05-13T05:06:00ZNishant VidyasagarUpgrading storage service for the core library dependencies# Merge request template# Merge request templateNishant VidyasagarNishant Vidyasagarhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/498Adding user id in cache key2023-05-13T05:06:00Zharshit aggarwalAdding user id in cache key# Merge request template# Merge request templateharshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/561Checking pipelines2023-05-13T05:05:57ZNishant VidyasagarChecking pipelines# Merge request template# Merge request templateNishant VidyasagarNishant Vidyasagarhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/549Draft: fix policy url2023-05-13T05:05:55ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comDraft: fix policy url# Merge request template# Merge request templatehttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/685Set FailOnError to false2023-05-09T18:48:49ZYash DholakiaSet FailOnError to falseSet FailOnError to false so build does not fail when dependency-check-maven fails.Set FailOnError to false so build does not fail when dependency-check-maven fails.M18 - Release 0.21Yash DholakiaYash Dholakiahttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/688making sure test for special character always run in aws2023-04-26T20:26:55ZAbhay Joshimaking sure test for special character always run in aws# Merge request template# Merge request templateM18 - Release 0.21Abhay JoshiAbhay Joshihttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/684making sure test for special character always run in aws2023-04-26T15:51:34ZAbhay Joshimaking sure test for special character always run in awsM18 - Release 0.21Okoun-Ola Fabien HouetoAbhay JoshiOkoun-Ola Fabien Houetohttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/681Trusted azure/cherrypick m14 m162023-04-21T19:38:00ZSwapnil PancholiTrusted azure/cherrypick m14 m16# Merge request template# Merge request templatehttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/680Merge branch 'aws-integration-eks' into 'master'2023-04-21T18:11:42ZSwapnil PancholiMerge branch 'aws-integration-eks' into 'master'https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/675Merge azure/cherrypick-m14-16 into azure/m16-master2023-04-21T18:10:58ZOsborn ChenMerge azure/cherrypick-m14-16 into azure/m16-master# Merge request template# Merge request templatehttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/676Remove SNAPSHOT dependencies2023-04-13T19:03:21ZDavid Diederichd.diederich@opengroup.orgRemove SNAPSHOT dependenciesThis automated MR removes usage of `SNAPSHOT` versions in the first party library dependencies.
Since `SNAPSHOT` dependencies change frequently -- by their nature -- usage of them across projects is dangerous and should be avoided.
### ...This automated MR removes usage of `SNAPSHOT` versions in the first party library dependencies.
Since `SNAPSHOT` dependencies change frequently -- by their nature -- usage of them across projects is dangerous and should be avoided.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: 181b245edcd31f730b8bfdb855e299c4e413161f
Maven: 0.21.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | --------------- | ---------------------- |
| core-lib-azure | 0.21.0-rc1 | |
| core-lib-gc | 0.21.0-rc1 | |
| os-core-lib-aws | 0.21.0-SNAPSHOT | 0.19.0-rc3 |
| obm | 0.20.0-rc2 | |
| oqm | 0.19.0 | |
| os-core-common | 0.20.0-rc3 | 0.13.0, 0.18.0-rc3 |
| os-core-lib-ibm | 0.17.0-rc4 | 0.17.0-rc4 |
| osm | 0.21.0-rc1 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.0 | 2.8.1, 2.13.2.2 |
| (3rd Party) org.springframework.spring-webmvc | 5.3.26, 5.3.22 | 5.3.12, 5.3.24, 5.3.22 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.27, 1.30 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade-2
SHA: c76288d84a1ce97e663476aea7e68f99d43a4540
Maven: 0.21.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | -------------- | ---------------------- |
| core-lib-azure | 0.21.0-rc1 | |
| core-lib-gc | 0.21.0-rc1 | |
| os-core-lib-aws | 0.20.0 | 0.19.0-rc3 |
| obm | 0.20.0-rc2 | |
| oqm | 0.19.0 | |
| os-core-common | 0.20.0-rc3 | 0.13.0, 0.18.0-rc3 |
| os-core-lib-ibm | 0.17.0-rc4 | 0.17.0-rc4 |
| osm | 0.21.0-rc1 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.0 | 2.8.1, 2.13.2.2 |
| (3rd Party) org.springframework.spring-webmvc | 5.3.26, 5.3.22 | 5.3.12, 5.3.24, 5.3.22 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.27, 1.30 |M17 - Release 0.20https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/668remove snakeyaml from dependencies2023-04-05T13:08:55ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comremove snakeyaml from dependencies# Merge request template# Merge request templatehttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/611AWS Merge2023-03-28T15:29:37ZAbhishek PatilAWS MergeSpring Security core version higher than 5.7.5 and other vulnerable package upgradesSpring Security core version higher than 5.7.5 and other vulnerable package upgradesM17 - Release 0.20Okoun-Ola Fabien HouetoYash DholakiaManish JangidOkoun-Ola Fabien Houetohttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/652Filter metadata records by partition id2023-03-16T20:45:06ZMorris EstepaFilter metadata records by partition id## Type of change
- [X] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
https://community.opengroup.org/osdu/platform/system/storage/-/issues/167
## Does this introduce a change in ...## Type of change
- [X] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
https://community.opengroup.org/osdu/platform/system/storage/-/issues/167
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
metadata records from other data partition ids show up when querying records for a given partition id
## What is the new/expected behavior?
only metadata records from the target partition id shows up in the query results
## Have you added/updated Unit Tests and Integration Tests?
Yes
## Any other useful informationM16 - Release 0.19Morris EstepaMorris Estepahttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/605Conversion: validate records at the start using json schemas2023-03-10T13:17:20ZMykyta SavchukConversion: validate records at the start using json schemasThere is a bug, when SpatialLocation in record is array, storage returns 500 on batch. Example record data:
```
"SpatialLocation": [
{
"SpatialGeometryTypeID": "Point",
....
]
```
And there were a lot of sim...There is a bug, when SpatialLocation in record is array, storage returns 500 on batch. Example record data:
```
"SpatialLocation": [
{
"SpatialGeometryTypeID": "Point",
....
]
```
And there were a lot of similar bugs related to record geo data validation lately (like misspelling field names and so on, that resulted in 500), but the existing validation logic is not intuitive and is spread across the code with random checks, so to fix them we had to add different checks (if's) in different parts of the code, and to make this logic more robust and prevent future bugs it was decided to do a refactoring and place the validation in one place using json schemas, since we already have them, and remove any further validation checks. Also, as we already have a well defined schema for the SpatialLocation geo attribute, I added a validation for such case in addition to the existing generic geo attribute validation, which I also converted to json schema (GenericGeoAttribute.json).
- Changed conversion logic to add validation against the schema at the start of the flow;
- Removed unnecessary validation checks;
- In case of validation error, the message from validation library is returned, instead of constant predefined messages;
- I changed tests' data (unit and integration), because the schema for AnyCrsFeatureCollection says that if bbox is present, it should not be null, so I added a sample value.M17 - Release 0.20Chad LeongMykyta SavchukChad Leong