Storage merge requestshttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests2023-08-18T22:21:20Zhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/417updated core-lib-ibm version to v0.15.0-rc22023-08-18T22:21:20ZBhushan Radeupdated core-lib-ibm version to v0.15.0-rc2# Merge request template# Merge request templateM12 - Release 0.15Bhushan RadeBhushan Radehttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/416GONRG-4953: fix deploy for anthos env2023-08-18T22:21:21ZAliaksandr Ramanovich (EPAM)GONRG-4953: fix deploy for anthos env# Merge request template
fix deploy for storage in anthos env# Merge request template
fix deploy for storage in anthos envM12 - Release 0.15Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/412update to pass partition id in opa request url2022-09-15T10:16:45ZKelly Zhouupdate to pass partition id in opa request url# Merge request template
Please refer to this page for the changes: https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/wikis/Support-Multi-Partition-Policies-in-OPA# Merge request template
Please refer to this page for the changes: https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/wikis/Support-Multi-Partition-Policies-in-OPAM12 - Release 0.15https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/422[GONRG-4953] Fix anthos pipeline2023-08-18T22:21:17ZDanylo Vanin (EPAM)[GONRG-4953] Fix anthos pipeline# Merge request template# Merge request templateM12 - Release 0.15https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/421fix token provisioning for anthos2022-05-25T15:51:00ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comfix token provisioning for anthos# Merge request template# Merge request templateM12 - Release 0.15Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/404update Storage API spec2023-08-18T22:21:33ZAn Ngoupdate Storage API spec## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
https://community.opengroup.org/osdu/platform/system/storage/-/issues/37
## Does this introduce a change in t...## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
https://community.opengroup.org/osdu/platform/system/storage/-/issues/37
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
The [Storage API spec](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/api/storage_openapi.yaml) does not match the implementation capabilities. Here the two issues:
* batch record read with normalization (as used by the indexer).
* patch capability for acl, legal and the likes, which does not bump the version.
Structural error at info.contact.email
should match format "email"
format: email
Jump to line 9
Structural error at paths./records/delete.post
should NOT have additional properties
additionalProperty: requestBody
Jump to line 200
Structural error at paths./records/delete.post.parameters.0
should NOT have additional properties
additionalProperty: schema
Jump to line 208
Structural error at paths./records/delete.post.parameters.0
should have required property 'type'
missingProperty: type
Jump to line 208
Structural error at paths./records/delete.post.responses.207
should NOT have additional properties
additionalProperty: content
Jump to line 225
Semantic error at paths./records/delete.post.responses.207.content.application/json.schema.items.$ref
$refs must reference a valid location in the document
Jump to line 232
Semantic error at paths./records/delete.post.security.0
Security requirements must match a security definition
Jump to line 238
Semantic error at paths./records/delete.post.security.1
Security requirements must match a security definition
Jump to line 239
Structural error at definitions.MultiRecordIds.properties.attributes
should NOT have additional properties
additionalProperty: allowEmptyValue
Jump to line 728
Structural error at definitions.Record.properties.history
should NOT have additional properties
additionalProperty: allowEmptyValue
Jump to line 849
Structural error at definitions.RecordHistory.properties.parents
should NOT have additional properties
additionalProperty: allowEmptyValue
Jump to line 875
Structural error at definitions.RecordBulkUpdateParam.properties.query
should NOT have additional properties
additionalProperty: allowEmptyValue
Jump to line 957
Structural error at definitions.RecordBulkUpdateParam.properties.ops
should NOT have additional properties
additionalProperty: allowEmptyValue
Jump to line 961
Semantic error at definitions.RecordQuery.properties.ids
Schemas with 'type: array', require a sibling 'items: ' field
Jump to line 996
## What is the new/expected behavior?
Update Storage API spec to include Batch record read with normalization; fixed swagger structural and semantic errors.
## Have you added/updated Unit Tests and Integration Tests?
N/A
## Any other useful information
N/AM12 - Release 0.15https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/431Common pipeline for preship env [GONRG-5050]2023-08-18T22:21:10ZMikhail Piatliou (EPAM)Common pipeline for preship env [GONRG-5050]# Merge request template# Merge request templateM13 - Release 0.16Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/433Gonrg 5061 refactor community pipeline2023-08-18T22:21:08ZSiarhei Symanovich (EPAM)Gonrg 5061 refactor community pipelineGonrg 5061 refactor community pipelineGonrg 5061 refactor community pipelineM13 - Release 0.16Siarhei Symanovich (EPAM)Siarhei Symanovich (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/427Bulk record creation API fix for MSI2023-08-18T22:21:12ZAnkur RawatBulk record creation API fix for MSICurrent Bulk Executor API used Document Client which is creeated with cosmos-primary-key.
To move to MSI, we need a client which can be created with DefaultCreds. Made changes for the same.Current Bulk Executor API used Document Client which is creeated with cosmos-primary-key.
To move to MSI, we need a client which can be created with DefaultCreds. Made changes for the same.M13 - Release 0.16Ankur RawatAnkur Rawathttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/426fix opa endpoint config (GONRG-5080)2022-09-10T09:09:15ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comfix opa endpoint config (GONRG-5080)## Type of change
- [x] Bug Fix
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Does th...## Type of change
- [x] Bug Fix
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Property configuration of OPA endpoint combined with OPA service not working as it was planned,
value does not inject to the service, and during runtime, a NullPointer exception will occur when Storage tries to send request OPA endpoint.
And it no follow best practices as it said that configuration should be clean POJO objects.
https://www.baeldung.com/configuration-properties-in-spring-boot#simple-properties
## What is the new/expected behavior?
Property injects as it should be, via property file or via env variable.
## Have you added/updated Unit Tests and Integration Tests?
yes
## Any other useful information
This is a minor fix that will not require any changes in CSP configurations.M13 - Release 0.16Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/434GONRG-5051: refactor dev2 pipeline2023-08-18T22:21:06ZNastassia Rabeichykava (EPAM)GONRG-5051: refactor dev2 pipeline# Merge request template# Merge request templateM13 - Release 0.16https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/435Upgrade Core IBM Library for Release 0.152022-06-17T15:36:58ZDavid Diederichd.diederich@opengroup.orgUpgrade Core IBM Library for Release 0.15This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any...This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any library that is older than the previous release will be left as-is, since the upgrade is likely to be more complicated.
Furthermore, the upgrade should only be merged in the CI pipeline reports success.
If this MR has failed, we can spend a little time investigating to see if a trivial upgrade could achieve compatiblity to the new library.
But significant upgrade efforts should not occur on this MR, as part of the release tagging process.
Instead, significant work should be scheduled for a subsequent milestone.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: 6d3c794b8cc82b76468007f7cd89c031f172c65f
Maven: 0.16.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | provider/storage-aws/src/main/ComplianceTrigger/ComplianceTriggerFunction/ComplianceTriggerFunction/ | testing/ |
| ------------------------------------------------------- | ------------------ | ---------------------------------------------------------------------------------------------------- | ----------------- |
| core-lib-azure | 0.14.0-rc5 | | |
| core-lib-gcp | 0.15.0-rc3, 0.13.0 | | |
| os-core-lib-aws | 0.14.0-rc2, 0.3.8 | 0.14.0-rc2 | 0.14.0-rc2, 0.3.8 |
| obm | 0.15.0-rc5 | | |
| oqm | 0.15.0-rc2 | | |
| os-core-common | 0.15.0-SNAPSHOT | 0.13.0 | 0.13.0 |
| os-core-lib-ibm | 0.15.0-rc2 | | 0.13.0 |
| osm | 0.15.0-rc5 | | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.13.2.2 | 2.6.7.2 | 2.8.1, 2.13.2.2 |
| (3rd Party) net.minidev.json-smart | 2.4.7 | | 2.3 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.13.3 | 2.17.1 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1 | 2.13.3 | 2.17.1 |
### Dependency Information After the Upgrade
```
Branch: core-ibm-upgrade
SHA: 912ac6e084d660c1d4475c6ada8f7566a4a97f0f
Maven: 0.16.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | provider/storage-aws/src/main/ComplianceTrigger/ComplianceTriggerFunction/ComplianceTriggerFunction/ | testing/ |
| ------------------------------------------------------- | ------------------ | ---------------------------------------------------------------------------------------------------- | ----------------- |
| core-lib-azure | 0.14.0-rc5 | | |
| core-lib-gcp | 0.15.0-rc3, 0.13.0 | | |
| os-core-lib-aws | 0.14.0-rc2, 0.3.8 | 0.14.0-rc2 | 0.14.0-rc2, 0.3.8 |
| obm | 0.15.0-rc5 | | |
| oqm | 0.15.0-rc2 | | |
| os-core-common | 0.15.0-SNAPSHOT | 0.13.0 | 0.13.0 |
| os-core-lib-ibm | 0.15.2 | | 0.13.0 |
| osm | 0.15.0-rc5 | | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.13.2.2 | 2.6.7.2 | 2.8.1, 2.13.2.2 |
| (3rd Party) net.minidev.json-smart | 2.4.7 | | 2.3 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.13.3 | 2.17.1 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1 | 2.13.3 | 2.17.1 |M12 - Release 0.15https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/418Remove read audit logs2023-08-18T22:21:18ZRostislav Vatolinvatolinrp@gmail.comRemove read audit logs## Type of change
- [YES] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/49
## Does this introduce a change i...## Type of change
- [YES] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/49
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
all read audit logs are being logged
## What is the new/expected behavior?
read audit logs can be turned off using entry (flag) in PartitionInfo table
## Have you added/updated Unit Tests and Integration Tests?
unit tests updatedM13 - Release 0.16https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/438Cherry-pick 'Upgrade Core IBM Library for Release 0.15' into release/0.152022-06-20T20:38:38ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Upgrade Core IBM Library for Release 0.15' into release/0.15Original MR: !435Original MR: !435M12 - Release 0.15David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/440bulk API update2022-06-23T14:46:31ZAnkur Rawatbulk API updateBulk API update to incorporate MSI changesBulk API update to incorporate MSI changesAnkur RawatAnkur Rawathttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/444updated helm chart2022-06-29T13:59:49ZBhushan Radeupdated helm chartHelm chart updated by IBM teamHelm chart updated by IBM teamBhushan RadeBhushan Radehttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/445GONRG-5228: Added .dockerignore and .gitattributes2023-08-18T22:21:03ZKrisztian Molnar (EPAM)GONRG-5228: Added .dockerignore and .gitattributesM13 - Release 0.16Krisztian Molnar (EPAM)Krisztian Molnar (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/446added logs for verifying cache invalidation2022-07-07T12:33:37ZAnkur Rawatadded logs for verifying cache invalidationAdded logs to verify the invalidation of cache for legal tag deletion. Also added sleep timer for 2 seconds before making create calls.Added logs to verify the invalidation of cache for legal tag deletion. Also added sleep timer for 2 seconds before making create calls.Ankur RawatAnkur Rawathttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/423implement int tests for anthos env(GONRG-4979)2023-08-18T22:21:15ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comimplement int tests for anthos env(GONRG-4979)# Merge request template# Merge request templateM13 - Release 0.16Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/450Remove Snapshot version from core-common- dependency2022-07-12T19:30:28ZShrikant GargRemove Snapshot version from core-common- dependency# Merge request template
https://community.opengroup.org/osdu/platform/system/storage/-/issues/135
Upgrading core-common as it is referring to SNAPSHOT version# Merge request template
https://community.opengroup.org/osdu/platform/system/storage/-/issues/135
Upgrading core-common as it is referring to SNAPSHOT versionM12 - Release 0.15Shrikant GargShrikant Garg