Storage merge requestshttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests2023-08-18T22:29:15Zhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/57upgrade springboot to fix whitesource vulnerability2023-08-18T22:29:15ZAliaksei Darafeyeuupgrade springboot to fix whitesource vulnerability# Merge request template# Merge request templateM1 - Release 0.1ethiraj krishnamanaiduNitin-slbNeelesh ThakurSherman YangYauheni Lesnikauethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/58Enable transaction logger and slf4j logger in storage-azure2023-08-18T22:29:14ZVibhuti Sharma [Microsoft]Enable transaction logger and slf4j logger in storage-azure**Description**
* Enabled slf4j logger and transaction logger by updating the version of os-core-lib in pom.xml file for storage-azure and adding the necessary commands in application.properties
* Added exclusions in `spring-boot-starter...**Description**
* Enabled slf4j logger and transaction logger by updating the version of os-core-lib in pom.xml file for storage-azure and adding the necessary commands in application.properties
* Added exclusions in `spring-boot-starter-security` dependency to enable slf4j logger binding without conflict
* Verified locally the reflection of test logs in `Traces` table in AppInsights
* \[YES\] Have you followed our code review [guidelines](https://github.com/microsoft/code-with-engineering-playbook/blob/master/pull-requests/code-reviews/readme.md)?
* \[YES\] Have you added an explanation of what your changes do and why you'd like us to include them?
* \[NO\] I have updated the documentation accordingly.
* \[NA\] I have added tests to cover my changes.
* \[YES\] All new and existing tests passed.
* \[YES\] My code follows the code style of this project.
* \[YES\] I ran lint checks locally prior to submission.M1 - Release 0.1Vibhuti Sharma [Microsoft]Vibhuti Sharma [Microsoft]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/59AWS bulk-acl implementation2020-09-04T16:19:18ZMatt WiseAWS bulk-acl implementation# Merge request template# Merge request templateM1 - Release 0.1Rucha DeshpandeMatt WiseRucha Deshpandehttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/61fix bootstrap issues2023-08-18T22:29:12ZAliaksei Darafeyeufix bootstrap issues# Merge request template# Merge request templateM1 - Release 0.1ethiraj krishnamanaiduSherman Yangethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/63Add osdu-gcp as a target environment for CICD2023-08-18T22:29:10ZOleksandr Kosse (EPAM)Add osdu-gcp as a target environment for CICD- Add osdu-gcp enviromentas a target for CICD
- Check integration tests for osdu-gcp environment- Add osdu-gcp enviromentas a target for CICD
- Check integration tests for osdu-gcp environmentM1 - Release 0.1David Diederichd.diederich@opengroup.orgDmitriy RudkoRostislav Dublin (EPAM)David Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/64Disabling AAD auth for storage service2023-08-18T22:29:08ZKiran VeerapaneniDisabling AAD auth for storage serviceAs Istio AuthN got enabled in R2 services,Disabling AAD authentication using Istio auth property. Added Istio auth Env variable and set to true in helm charts templateAs Istio AuthN got enabled in R2 services,Disabling AAD authentication using Istio auth property. Added Istio auth Env variable and set to true in helm charts templateM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/66Various namespaces in datastore (GONRG-837)2023-08-18T22:29:07ZRiabokon Stanislav(EPAM)[GCP]Various namespaces in datastore (GONRG-837)## Type of change
- [X] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] IB...## Type of change
- [X] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Current version of Storage service in Community (deployed from Opengroup GitLab) works with default namespace. But all data, uploaded with storage service from EpamGit (d773a274 version), stored in osdu namespace.
## What is the new/expected behavior?
* Changed a version of GCP-LIB to 0.3.22
* Namespace configuration with TenantInfo in DatastoreFactory
* Separate configuration, for configurable namespace and default
Storage service can work with all namespaces in Datastore.
## Have you added/updated Unit Tests and Integration Tests?
Integration Tests cover it
## Any other useful informationM1 - Release 0.1Dmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/68New storage azure2023-08-18T22:29:05ZChristian LecknerNew storage azure# Merge request template# Merge request templateM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/72Fixed KV env variable2023-08-18T22:28:52ZDaniel SchollFixed KV env variableSimple Env Name ChangeSimple Env Name ChangeM1 - Release 0.1Daniel SchollDaniel Schollhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/73Consume Partition service to provide multi-tenancy support in Storage-Azure2023-08-18T22:28:24ZAlok JoshiConsume Partition service to provide multi-tenancy support in Storage-Azure## Type of change
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementat...## Type of change
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] Azure
## Does this introduce a breaking change?
- [YES]
## What is the current behavior?
Current Azure implementation works in single tenancy mode. This implementation aims to provide multi-tenancy support via Partition service
## What is the new/expected behavior?
Provides multi-tenancy support via Partition service
## Have you added/updated Unit Tests and Integration Tests?
Yes
## Any other useful informationM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/74Fix issue with persistable reference handling in converters;2023-08-18T22:27:34ZYauheni LesnikauFix issue with persistable reference handling in converters;This merge request is link with the another one in [os-core-common](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/23).
The issue was found in the **Storage GA** and concerned feature of the...This merge request is link with the another one in [os-core-common](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/23).
The issue was found in the **Storage GA** and concerned feature of the **Gson** lib, namely `getAsString()` method of the `com.google.gson.JsonElement` class with the subclasses (**JsonPrimitive**, **JsonObject**, etc). The thing is that the method overriden safely only for **JsomPrimitive** subclass, and if the field mapped as **JsonObject** one (persistableReference in the 'meta' part, in our case) this call throws **UnsupportedOperationException**.
My enhancement prevents undesirable throwing the exception. Also tests were extendedM1 - Release 0.1ethiraj krishnamanaiduFerris ArgyleDania Kodeih (Microsoft)Wladmir FrazaoJoeNitin-slbNeelesh ThakurKishore BattulaMatt WiseYauheni Lesnikauethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/76Rollback datastoreFactory.getDatastoreDefaultNamespace() (GONRG-837)2023-08-18T22:27:32ZRiabokon Stanislav(EPAM)[GCP]Rollback datastoreFactory.getDatastoreDefaultNamespace() (GONRG-837)## Type of change
- [X] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] IB...## Type of change
- [X] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Current version of Storage service in Community (deployed from Opengroup GitLab) works with default namespace. But all data, uploaded with storage service from EpamGit (d773a274 version), stored in osdu namespace.
`this.datastoreFactory.getDatastore();`
`public Datastore getDatastore() {
DatastoreCredential credential = getDatastoreCredential();
return DatastoreOptions.newBuilder()
.setRetrySettings(RETRY_SETTINGS)
.setTransportOptions(TRANSPORT_OPTIONS)
.setCredentials(credential)
.setProjectId(this.tenant.getProjectId())
.setNamespace(this.tenant.getName())
.build().getService();
}
`
## What is the new/expected behavior?
* Namespace configuration with TenantInfo in DatastoreFactory
Storage service can work with all namespaces in Datastore.
## Have you added/updated Unit Tests and Integration Tests?
## Any other useful informationM1 - Release 0.1Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/77Re-integrating storage service with Azure Monitor2023-08-18T22:27:31ZVibhuti Sharma [Microsoft]Re-integrating storage service with Azure Monitor## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NO] I have updated the documentation accordingly.
* [NA] I have added tests t...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NO] I have updated the documentation accordingly.
* [NA] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [YES] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
High level design:
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
Due to azure.version being updated to 2.1.10, logs stopped getting populated in Traces table in AppInsights. This is because there is a bug in azure-spring-boot version 2.1.10, an issue regarding the same has been created here - [link](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-sdk-for-java%2Fissues%2F15773&data=02%7C01%7CVibhuti.Sharma%40microsoft.com%7C8d7840419e7f4e4d6d3a08d86612dadc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637371578488845944&sdata=dbB0Q9rgzFuNwawFatYjuMiI6qPeUdNsq86XIsvOJu8%3D&reserved=0)
By updating to azure-spring-boot version >= 2.2.0, we can see logs in appinsights again.
Have also enabled MDC context filter
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
- updated azure-spring-boot dependency in pom file
- updated core-lib-azure version
- enabled mdc context filter
- updated few other dependencies to resolve errors
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
N/A
## Does this introduce a breaking change?
-------------------------------------
- [NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
N/A
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
N/A
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M1 - Release 0.1Vibhuti Sharma [Microsoft]Vibhuti Sharma [Microsoft]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/78Storage azure mod2023-08-18T22:27:29ZChristian LecknerStorage azure mod# Merge request template# Merge request templateM1 - Release 0.1Christian LecknerChristian Lecknerhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/79GCP implement index cleanup (GONRG-856)2023-08-18T22:27:27ZIgor Filippov (EPAM)GCP implement index cleanup (GONRG-856)## Type of change
- [ ] Bug Fix
- [X] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/5#note_16693
## Does this intro...## Type of change
- [ ] Bug Fix
- [X] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/5#note_16693
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
GCP provider don't send information in indexer-service about schema deletes.
## What is the new/expected behavior?
GCP provider send information in indexer-service about schema deletes
## Have you added/updated Unit Tests and Integration Tests?
- [NO]
## Any other useful information
Changes:
- Add publish message in SchemaServiceImpl class, when schema deletesM1 - Release 0.1Dmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/80Use Redis cache for Schema2023-08-18T22:27:26ZAlok JoshiUse Redis cache for Schema# Merge request template
Use centralized Redis cache instead of VM cache for Schemas in Storage service. This is a change to address this bug https://community.opengroup.org/osdu/platform/system/storage/-/issues/34. App only uses VM cach...# Merge request template
Use centralized Redis cache instead of VM cache for Schemas in Storage service. This is a change to address this bug https://community.opengroup.org/osdu/platform/system/storage/-/issues/34. App only uses VM cache when running locally because setting up a Redis instance with SSL is tricky.M1 - Release 0.1Neelesh ThakurNeelesh Thakurhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/81Audit events for storage (GONRG-941)2023-08-18T22:27:24ZRiabokon Stanislav(EPAM)[GCP]Audit events for storage (GONRG-941)## Type of change
- [ ] Bug Fix
- [X] Feature
https://community.opengroup.org/osdu/platform/deployment-and-operations/audit-and-metrics/-/issues/15
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a c...## Type of change
- [ ] Bug Fix
- [X] Feature
https://community.opengroup.org/osdu/platform/deployment-and-operations/audit-and-metrics/-/issues/15
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [X] Azure
- [X] GCP
- [X] IBM
## Does this introduce a breaking change?
- [YES]
## What is the current behavior?
This event is logged with a hard-coded phrase "All Kinds"
## What is the new/expected behavior?
This event is logged with all kinds.
for example:
`opendes:bulkupdate:test:1.1.1599131561064, opendes:bulkupdate:test:1.1.1599132153171, opendes:bulkupdate:test:1.1.1599132483332, opendes:bulkupdate:test:1.1.1599132676337, opendes:bulkupdate:test:1.1.1599133046031, opendes:bulkupdate:test:1.1.1599133394096, opendes:bulkupdate:test:1.1.1599133437936, opendes:bulkupdate:test:1.1.1599133526479, opendes:bulkupdate:test:1.1.1602591024552, opendes:bulkupdate:test:1.1.1602591910335`
## Have you added/updated Unit Tests and Integration Tests?
- [YES]
## Any other useful informationM1 - Release 0.1Dmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/82migrate storage service from AppEngine to CloudRun GONRG-9732023-08-18T22:27:22ZMykola Zamkovyi (EPAM)migrate storage service from AppEngine to CloudRun GONRG-973# Merge request template# Merge request templateM1 - Release 0.1Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/84adds recordIdVersions to createOrUpdateRecords RS2023-08-18T22:27:20ZAliaksei Darafeyeuadds recordIdVersions to createOrUpdateRecords RS## Type of change
- [X] Feature
https://community.opengroup.org/osdu/documentation/-/issues/73
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so whic...## Type of change
- [X] Feature
https://community.opengroup.org/osdu/documentation/-/issues/73
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [NO]
## Does this introduce a breaking change?
- [NO]
## What is the new/expected behavior?
adds new field to CreateUpdateRecordsResponseM4 - Release 0.7ethiraj krishnamanaiduNitin-slbNeelesh ThakurDaniel SchollPavel Bachylaashley kelhamDuvelis CaraoYauheni Lesnikauethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/86Removed unhardcode repositories and distribution management from all poms (GO...2023-08-18T22:27:19ZRiabokon Stanislav(EPAM)[GCP]Removed unhardcode repositories and distribution management from all poms (GONRG-993)## Type of change
- [ ] Bug Fix
- [X] Feature
ADR: https://community.opengroup.org/osdu/platform/system/home/-/issues/55
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider...## Type of change
- [ ] Bug Fix
- [X] Feature
ADR: https://community.opengroup.org/osdu/platform/system/home/-/issues/55
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [X] Azure
- [X] GCP
- [X] IBM
## Does this introduce a breaking change?
- [YES]
## What is the new/expected behavior?
This MR brings changes to POMs (root, azure, gcp) to improve (unhardcode) the way how repositories are being pointed. It now uses set of properties defined in the `.mvn/community-maven.settings.xml`
## Have you added/updated Unit Tests and Integration Tests?
- [YES]
## Any other useful informationM1 - Release 0.1Dmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudko