Storage merge requestshttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests2023-03-23T18:55:00Zhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/644Spatial location array2023-03-23T18:55:00ZMykyta SavchukSpatial location arrayWhen SpatialLocation is an array in a record, storage batch API with conversion returns 500. To fix this, I added a check if the geo attribute is a json object.When SpatialLocation is an array in a record, storage batch API with conversion returns 500. To fix this, I added a check if the geo attribute is a json object.M17 - Release 0.20Mykyta SavchukMykyta Savchukhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/643Added logs for not found records2023-03-15T09:32:53ZDmitrii Novikov (EPAM)Added logs for not found records## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] I...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Added logs for `not found` records: due to missed metadata, missed during read from cloud storage and missded during conversion.
Some records were missed during indexing processM17 - Release 0.20Dmitrii Novikov (EPAM)Dmitrii Novikov (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/642Added modify user and time info2023-09-08T19:26:41Zvinay yadavAdded modify user and time info**Implementation of original ADR https://community.opengroup.org/osdu/platform/system/storage/-/issues/148**
# Merge request template**Implementation of original ADR https://community.opengroup.org/osdu/platform/system/storage/-/issues/148**
# Merge request templateM17 - Release 0.20https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/641Cherry pick increased memory limits change to release/0.192023-03-10T09:28:46ZMorris EstepaCherry pick increased memory limits change to release/0.19Original merge https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/640Original merge https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/640M16 - Release 0.19David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/640Increase memory limits to 1.8GB2023-03-10T09:28:47ZMorris EstepaIncrease memory limits to 1.8GB## Type of change
- [X] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provi...## Type of change
- [X] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Memory limit is at 900MB
## What is the new/expected behavior?
Memory limit is at 1.8GB
## Have you added/updated Unit Tests and Integration Tests?
No
## Any other useful informationM16 - Release 0.19Morris EstepaMorris Estepahttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/639Extend patch API to support data block modification2023-09-08T19:26:41ZAlok JoshiExtend patch API to support data block modificationThis change is an effort to implement [ADR](https://community.opengroup.org/osdu/platform/system/storage/-/issues/124).
Some considerations:
- This feature is implemented as a new API. To comply with the [RFC spec](https://www.rfc-edit...This change is an effort to implement [ADR](https://community.opengroup.org/osdu/platform/system/storage/-/issues/124).
Some considerations:
- This feature is implemented as a new API. To comply with the [RFC spec](https://www.rfc-editor.org/rfc/rfc6902.html), API media type must be "application/json-patch+json". The new API uses the same request mapping and HTTP method as the existing bulk update API
- Support for updating data block (i.e. 'data' and 'meta' fields in Record) is added. We are also extending metadata update support for additional fields (ancestry/parents, kind)
- There is no update to storage_openapi.yaml file under /docs/api because the swagger is generated with code now.
Active work items:
- Update tutorial documentation
Note to reviewers:
- There are a lot of seemingly unrelated small unit test changes. We made those because of a change in the mockito dependency, which performs strict mock stubbing checks by default. So, we've relaxed those checks by default in the unit test classes. The mockito dependency change was made because we wanted to introduce WebMvcTest with SpringRunner for the new API. This helps us mock the http call to the new endpointM17 - Release 0.20Alok JoshiChad LeongAlok Joshihttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/638Corrected Mock parameters to fix tests2023-03-06T02:40:51ZAnkur RawatCorrected Mock parameters to fix testsUnit test fixUnit test fixAnkur RawatAnkur Rawathttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/637Cherry-pick 'Adding EDS DMS to Allowed Principals' into release/0.192023-03-03T08:35:58ZMarc Burnie [AWS]Cherry-pick 'Adding EDS DMS to Allowed Principals' into release/0.19Original MR: https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/636
AWS DevOps/infrastructure change only.
Bug fix for unreported issue for AWS EDS DMS service.Original MR: https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/636
AWS DevOps/infrastructure change only.
Bug fix for unreported issue for AWS EDS DMS service.M16 - Release 0.19David Diederichd.diederich@opengroup.orgMarc Burnie [AWS]David Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/636Adding EDS DMS to Allowed Principals2023-03-02T13:55:39ZMarc Burnie [AWS]Adding EDS DMS to Allowed PrincipalsAWS Infrastructure change only.AWS Infrastructure change only.M16 - Release 0.19Marc Burnie [AWS]Marc Burnie [AWS]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/635Cherry-pick: GONRG-6646: Add variable for disabling OPA2023-03-01T21:28:13ZVolodymyr Pienskoi [EPAM / GCP]Cherry-pick: GONRG-6646: Add variable for disabling OPAGONRG-6646: Add variable for disabling OPA
See merge request !633
(cherry-picked from commit 91f0f5ff)
9b86fa5c GONRG-6646: Add variable for disabling OPAGONRG-6646: Add variable for disabling OPA
See merge request !633
(cherry-picked from commit 91f0f5ff)
9b86fa5c GONRG-6646: Add variable for disabling OPAM16 - Release 0.19Volodymyr Pienskoi [EPAM / GCP]Volodymyr Pienskoi [EPAM / GCP]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/634remove not necessary content header requirement2023-03-08T11:22:09ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comremove not necessary content header requirement# Description:
GET query/records endpoint requires `Content-Type` header in the request. <br/>
~~~
curl --location 'https://community.gcp.gnrg-osdu.projects.epam.com/api/storage/v2/query/records?kind=osdu%3Awks%3Amaster-data--Well%3A1.0...# Description:
GET query/records endpoint requires `Content-Type` header in the request. <br/>
~~~
curl --location 'https://community.gcp.gnrg-osdu.projects.epam.com/api/storage/v2/query/records?kind=osdu%3Awks%3Amaster-data--Well%3A1.0.0' \
--header 'data-partition-id: osdu' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer
~~~
If a header is not provided Storage responds with a `415 Unsupported Media Type` code. <br/>
We need to remove this requirement since this endpoint doesn't accept request body, and shouldn't specify `content-type`. <br/>
# How to test:
GET query/records works without `Content-Type` header.
~~~
curl --location 'https://community.gcp.gnrg-osdu.projects.epam.com/api/storage/v2/query/records?kind=osdu%3Awks%3Amaster-data--Well%3A1.0.0' \
--header 'data-partition-id: osdu' \
--header 'Authorization: Bearer
~~~
# Changes include:
- [x] Bugfix (a non-breaking change that solves an issue).
# Changes in:
- [x] Common codeM17 - Release 0.20Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/633GONRG-6646: Add variable for disabling OPA2023-03-07T09:32:34ZVolodymyr Pienskoi [EPAM / GCP]GONRG-6646: Add variable for disabling OPAAdded `opaEnabled` value to Helm chart that disables OPA by default.Added `opaEnabled` value to Helm chart that disables OPA by default.M16 - Release 0.19Volodymyr Pienskoi [EPAM / GCP]Volodymyr Pienskoi [EPAM / GCP]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/632Renamed legaltags-changed topic (GONRG-6561)2023-03-06T10:28:17ZDmitrii Novikov (EPAM)Renamed legaltags-changed topic (GONRG-6561)Renamed legaltags-changed topicRenamed legaltags-changed topicM17 - Release 0.20Dmitrii Novikov (EPAM)Dmitrii Novikov (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/631Removed groupCache2023-02-28T08:40:37ZRiabokon Stanislav(EPAM)[GCP]Removed groupCache## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IB...## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Removed groupCacheM17 - Release 0.20Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/630update core-lib, work with nullables from datastore2023-02-24T17:12:20ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comupdate core-lib, work with nullables from datastore# Merge request template# Merge request templateM17 - Release 0.20Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/629Cherry-pick 'Bypass info and swagger apis in collaboration filter' into relea...2023-02-17T17:40:13ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Bypass info and swagger apis in collaboration filter' into release/0.19**Original MR**: !626
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !626
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/storage/-/pipelines/new?ref=cherry-pick-for-626)M16 - Release 0.19David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/628Cherry-pick 'Generate Swagger using springdoc-openapi' into release/0.192023-02-17T17:42:57ZChad LeongCherry-pick 'Generate Swagger using springdoc-openapi' into release/0.19**Original MR**: !622
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !622
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/storage/-/pipelines/new?ref=cherry-pick-for-622)M16 - Release 0.19David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/627Draft: Sk/fix authpolicy for swaggerapi2023-08-25T22:28:44Zshivani karipeDraft: Sk/fix authpolicy for swaggerapi# Merge request template# Merge request templatehttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/626Bypass info and swagger apis in collaboration filter2023-02-17T17:39:10ZAlok JoshiBypass info and swagger apis in collaboration filterCollaborationFilter, when enabled with data partition feature flag strategy, makes a call to Partition service. This call requires data-partition-id header, which is not passed to certain apis (info, swagger, health, etc), thus the api c...CollaborationFilter, when enabled with data partition feature flag strategy, makes a call to Partition service. This call requires data-partition-id header, which is not passed to certain apis (info, swagger, health, etc), thus the api call fails.
We can short-circuit the CollaborationFilter class when url contains one of theseM16 - Release 0.19Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/625Security Upgrade AWS2023-03-30T08:07:19ZYash DholakiaSecurity Upgrade AWS# Merge request template# Merge request templateM17 - Release 0.20