Storage merge requestshttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests2023-09-08T19:26:43Zhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/651impl get records metadata in one batch2023-09-08T19:26:43ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comimpl get records metadata in one batch# Description:
Improved fetch records request via improvement of OSM lib and new functionality that supports `IN` filters.
Instead of N+ separate request to DB now a single request may be used with multiple values.
# Changes include:
...# Description:
Improved fetch records request via improvement of OSM lib and new functionality that supports `IN` filters.
Instead of N+ separate request to DB now a single request may be used with multiple values.
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [ ] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GC
- [ ] Azure
- [ ] AWS
- [ ] IBM
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [x] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M18 - Release 0.21Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/650Handling data inconsistency2023-03-16T14:23:20ZDmitrii Novikov (EPAM)Handling data inconsistency
## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] I...
## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Fixed exception message and status codeM17 - Release 0.20Dmitrii Novikov (EPAM)Dmitrii Novikov (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/649Add terraform switch parameter for MongoDB2023-03-16T13:24:34ZAleh Shubko [EPAM]Add terraform switch parameter for MongoDBAWS DevOps Changes OnlyAWS DevOps Changes OnlyM17 - Release 0.20Aleh Shubko [EPAM]Marc Burnie [AWS]Aleh Shubko [EPAM]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/648Added logs for not found records2023-03-15T16:41:12ZDmitrii Novikov (EPAM)Added logs for not found records## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] I...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Added logs for `not found` records: due to missed metadata, missed during read from cloud storage and missded during conversion.
Some records were missed during indexing processM17 - Release 0.20Dmitrii Novikov (EPAM)Dmitrii Novikov (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/647Revert "Merge branch 'add-logs' into 'master'"2023-08-18T22:20:34ZRiabokon Stanislav(EPAM)[GCP]Revert "Merge branch 'add-logs' into 'master'"# Merge request template# Merge request templateM17 - Release 0.20https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/646Increase AWS container memory2023-03-10T18:59:10ZManish JangidIncrease AWS container memory# Merge request template
Increase AWS container memory
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/645# Merge request template
Increase AWS container memory
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/645M16 - Release 0.19https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/645Allocating more memory to container than JVM heap memory2023-03-10T18:24:54ZManish JangidAllocating more memory to container than JVM heap memory# Merge request template
Updating container memory# Merge request template
Updating container memoryM16 - Release 0.19https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/644Spatial location array2023-03-23T18:55:00ZMykyta SavchukSpatial location arrayWhen SpatialLocation is an array in a record, storage batch API with conversion returns 500. To fix this, I added a check if the geo attribute is a json object.When SpatialLocation is an array in a record, storage batch API with conversion returns 500. To fix this, I added a check if the geo attribute is a json object.M17 - Release 0.20Mykyta SavchukMykyta Savchukhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/643Added logs for not found records2023-03-15T09:32:53ZDmitrii Novikov (EPAM)Added logs for not found records## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] I...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Added logs for `not found` records: due to missed metadata, missed during read from cloud storage and missded during conversion.
Some records were missed during indexing processM17 - Release 0.20Dmitrii Novikov (EPAM)Dmitrii Novikov (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/642Added modify user and time info2023-09-08T19:26:41Zvinay yadavAdded modify user and time info**Implementation of original ADR https://community.opengroup.org/osdu/platform/system/storage/-/issues/148**
# Merge request template**Implementation of original ADR https://community.opengroup.org/osdu/platform/system/storage/-/issues/148**
# Merge request templateM17 - Release 0.20https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/641Cherry pick increased memory limits change to release/0.192023-03-10T09:28:46ZMorris EstepaCherry pick increased memory limits change to release/0.19Original merge https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/640Original merge https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/640M16 - Release 0.19David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/640Increase memory limits to 1.8GB2023-03-10T09:28:47ZMorris EstepaIncrease memory limits to 1.8GB## Type of change
- [X] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provi...## Type of change
- [X] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Memory limit is at 900MB
## What is the new/expected behavior?
Memory limit is at 1.8GB
## Have you added/updated Unit Tests and Integration Tests?
No
## Any other useful informationM16 - Release 0.19Morris EstepaMorris Estepahttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/639Extend patch API to support data block modification2023-09-08T19:26:41ZAlok JoshiExtend patch API to support data block modificationThis change is an effort to implement [ADR](https://community.opengroup.org/osdu/platform/system/storage/-/issues/124).
Some considerations:
- This feature is implemented as a new API. To comply with the [RFC spec](https://www.rfc-edit...This change is an effort to implement [ADR](https://community.opengroup.org/osdu/platform/system/storage/-/issues/124).
Some considerations:
- This feature is implemented as a new API. To comply with the [RFC spec](https://www.rfc-editor.org/rfc/rfc6902.html), API media type must be "application/json-patch+json". The new API uses the same request mapping and HTTP method as the existing bulk update API
- Support for updating data block (i.e. 'data' and 'meta' fields in Record) is added. We are also extending metadata update support for additional fields (ancestry/parents, kind)
- There is no update to storage_openapi.yaml file under /docs/api because the swagger is generated with code now.
Active work items:
- Update tutorial documentation
Note to reviewers:
- There are a lot of seemingly unrelated small unit test changes. We made those because of a change in the mockito dependency, which performs strict mock stubbing checks by default. So, we've relaxed those checks by default in the unit test classes. The mockito dependency change was made because we wanted to introduce WebMvcTest with SpringRunner for the new API. This helps us mock the http call to the new endpointM17 - Release 0.20Alok JoshiChad LeongAlok Joshihttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/638Corrected Mock parameters to fix tests2023-03-06T02:40:51ZAnkur RawatCorrected Mock parameters to fix testsUnit test fixUnit test fixAnkur RawatAnkur Rawathttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/637Cherry-pick 'Adding EDS DMS to Allowed Principals' into release/0.192023-03-03T08:35:58ZMarc Burnie [AWS]Cherry-pick 'Adding EDS DMS to Allowed Principals' into release/0.19Original MR: https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/636
AWS DevOps/infrastructure change only.
Bug fix for unreported issue for AWS EDS DMS service.Original MR: https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/636
AWS DevOps/infrastructure change only.
Bug fix for unreported issue for AWS EDS DMS service.M16 - Release 0.19David Diederichd.diederich@opengroup.orgMarc Burnie [AWS]David Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/636Adding EDS DMS to Allowed Principals2023-03-02T13:55:39ZMarc Burnie [AWS]Adding EDS DMS to Allowed PrincipalsAWS Infrastructure change only.AWS Infrastructure change only.M16 - Release 0.19Marc Burnie [AWS]Marc Burnie [AWS]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/635Cherry-pick: GONRG-6646: Add variable for disabling OPA2023-03-01T21:28:13ZVolodymyr Pienskoi [EPAM / GCP]Cherry-pick: GONRG-6646: Add variable for disabling OPAGONRG-6646: Add variable for disabling OPA
See merge request !633
(cherry-picked from commit 91f0f5ff)
9b86fa5c GONRG-6646: Add variable for disabling OPAGONRG-6646: Add variable for disabling OPA
See merge request !633
(cherry-picked from commit 91f0f5ff)
9b86fa5c GONRG-6646: Add variable for disabling OPAM16 - Release 0.19Volodymyr Pienskoi [EPAM / GCP]Volodymyr Pienskoi [EPAM / GCP]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/634remove not necessary content header requirement2023-03-08T11:22:09ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comremove not necessary content header requirement# Description:
GET query/records endpoint requires `Content-Type` header in the request. <br/>
~~~
curl --location 'https://community.gcp.gnrg-osdu.projects.epam.com/api/storage/v2/query/records?kind=osdu%3Awks%3Amaster-data--Well%3A1.0...# Description:
GET query/records endpoint requires `Content-Type` header in the request. <br/>
~~~
curl --location 'https://community.gcp.gnrg-osdu.projects.epam.com/api/storage/v2/query/records?kind=osdu%3Awks%3Amaster-data--Well%3A1.0.0' \
--header 'data-partition-id: osdu' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer
~~~
If a header is not provided Storage responds with a `415 Unsupported Media Type` code. <br/>
We need to remove this requirement since this endpoint doesn't accept request body, and shouldn't specify `content-type`. <br/>
# How to test:
GET query/records works without `Content-Type` header.
~~~
curl --location 'https://community.gcp.gnrg-osdu.projects.epam.com/api/storage/v2/query/records?kind=osdu%3Awks%3Amaster-data--Well%3A1.0.0' \
--header 'data-partition-id: osdu' \
--header 'Authorization: Bearer
~~~
# Changes include:
- [x] Bugfix (a non-breaking change that solves an issue).
# Changes in:
- [x] Common codeM17 - Release 0.20Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/632Renamed legaltags-changed topic (GONRG-6561)2023-03-06T10:28:17ZDmitrii Novikov (EPAM)Renamed legaltags-changed topic (GONRG-6561)Renamed legaltags-changed topicRenamed legaltags-changed topicM17 - Release 0.20Dmitrii Novikov (EPAM)Dmitrii Novikov (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/631Removed groupCache2023-02-28T08:40:37ZRiabokon Stanislav(EPAM)[GCP]Removed groupCache## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IB...## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Removed groupCacheM17 - Release 0.20Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]