Storage merge requestshttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests2023-08-18T22:25:28Zhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/114OSDU-GCP: parametrize containerization stage2023-08-18T22:25:28ZAliaksandr Ramanovich (EPAM)OSDU-GCP: parametrize containerization stageparametrize containerization stage to remove hardcoded images pathsparametrize containerization stage to remove hardcoded images pathsM1 - Release 0.1Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/115Publish changes from Storage (GONRG-1476)2023-08-18T22:25:26ZArtem Dobrynin (EPAM)Publish changes from Storage (GONRG-1476)## Type of change
- [x] Bug Fix
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provi...## Type of change
- [x] Bug Fix
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Currently, we do not have an ability to access GCS as a service account. The only possible behaviour is to log in as the user, so there were some issues connected with that.
## What is the new/expected behavior?
Added ability to switch on and off impersonalization (access GCS with current user or with service account)
## Have you added/updated Unit Tests and Integration Tests?
Updated tests for Google Cloud Storage
## Any other useful information
This Merge Request should be merged after `core-lib-gcp` changes ([link](https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/os-core-lib-gcp/-/merge_requests/9))M1 - Release 0.1Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/118Decouple kind and tenant2023-08-18T22:25:25ZSviatoslav NekhaienkoDecouple kind and tenantCheck of a Data Partition ID has been removed from a Kind.
Please see https://community.opengroup.org/osdu/platform/system/storage/-/issues/42 for details.Check of a Data Partition ID has been removed from a Kind.
Please see https://community.opengroup.org/osdu/platform/system/storage/-/issues/42 for details.M1 - Release 0.1https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/119Record Id and Kind Id changes2023-08-18T22:25:23Zethiraj krishnamanaiduRecord Id and Kind Id changesTo support R3 Schema we need to relax the Record Id and Kind Id validation.
Please see more details in [Relax Id](https://community.opengroup.org/osdu/platform/system/storage/-/issues/26)
/platform/system/storage/-/issues/26To support R3 Schema we need to relax the Record Id and Kind Id validation.
Please see more details in [Relax Id](https://community.opengroup.org/osdu/platform/system/storage/-/issues/26)
/platform/system/storage/-/issues/26M1 - Release 0.1ethiraj krishnamanaiduDania Kodeih (Microsoft)Wladmir FrazaoJoeThomas Gehrmann [slb]Dmitriy Rudkoethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/120Changing the send load of event grid2023-08-18T22:25:22ZKomal MakkarChanging the send load of event grid## All Submissions:
-------------------------------------
* [YES I have added an explanation of what changes in this merge do and why we should include it?
* [YES] Does the MR contain pipeline/ helm chart related changes?
* [YES] I have ...## All Submissions:
-------------------------------------
* [YES I have added an explanation of what changes in this merge do and why we should include it?
* [YES] Does the MR contain pipeline/ helm chart related changes?
* [YES] I have updated the documentation accordingly.
* [YES] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [YES] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
High level design:
The Event Grid has the capability to publish messages in array. This is likely to impact the performance of Indexer Service as a single storage put job will be sent to indexer going forward.
Eliminating the performance risk, changed the code to follow the style of Service Bus.
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO]
NO
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M1 - Release 0.1https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/121adding MongoDB support for AWS2023-08-18T22:25:20ZAleh Shubko [EPAM]adding MongoDB support for AWSAdded initial MongoDB implementation for AWSAdded initial MongoDB implementation for AWSM3 - Release 0.5Aleh Shubko [EPAM]Aleh Shubko [EPAM]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/123Cloud-agnostic module `storage-reference` for Anthos (GONRG-1423)2023-08-18T22:25:18ZRiabokon Stanislav(EPAM)[GCP]Cloud-agnostic module `storage-reference` for Anthos (GONRG-1423)# Description:
Added a new cloud-agnostic module `storage-reference`. This module can be used for Anthos GCP.
Database: MongoDB
Messaging Broker: RabbitMQ
# How to test:
Bussines cases.
# Changes include:
- [ ] Refactor (a non-break...# Description:
Added a new cloud-agnostic module `storage-reference`. This module can be used for Anthos GCP.
Database: MongoDB
Messaging Broker: RabbitMQ
# How to test:
Bussines cases.
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [X] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
# Dev Checklist:
- [x] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M1 - Release 0.1Dmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/124Azure adding cosmos bulk operations2023-08-18T22:25:16ZJasonAzure adding cosmos bulk operations## Type of change
- [ ] Bug Fix
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
N/A
## Does this introduce a change in the core logic?
- [YES/**NO**]
## Does this introduce a change in the ...## Type of change
- [ ] Bug Fix
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
N/A
## Does this introduce a change in the core logic?
- [YES/**NO**]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [YES/**NO**]
## What is the current behavior?
- For any batch size, each record is queried in Cosmos individually to see if it exists
- For any batch size, the records are uploaded one at a time to Cosmos
- Old unused pipelines are in the service repo
## What is the new/expected behavior?
- An environment variable determines the minimum batch size to use the DocumentBulkExecutor to upload the records in parallel. If the batch size is >= this minimum, the records will be uploaded in parallel using bulk executor. If the batch size is less than the minimum, the records are uploaded in serial as they were before.
- For all batch sizes, records are queried using a SQL query of the form `SELECT * FROM c WHERE c.id IN [list_of_ids_here]` to reduce time to query for large batch sizes.
- Old pipelines are deleted
## Have you added/updated Unit Tests and Integration Tests?
## Any other useful information
- MR adding bulk executor to core-lib-azure can be found [here](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/66).M1 - Release 0.1JasonJasonhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/126Adding CustomThreadPoolFactory2023-08-18T22:25:15ZVibhuti Sharma [Microsoft]Adding CustomThreadPoolFactory## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NO] Does the MR contain pipeline/ helm chart related changes?
* [NO] I have u...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NO] Does the MR contain pipeline/ helm chart related changes?
* [NO] I have updated the documentation accordingly.
* [NA] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [NA] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
High level design:
Adding CustomThreadPoolFactory to enable creation of `CustomThreadPoolExecutor` from core-lib-azure, which allows for copying of context from main thread to child threads.
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
Concurrency classes being used from the java.util.concurrency package do not allow copying of MDC and request context from main thread to child threads.
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
## Does this introduce a breaking change?
-------------------------------------
- NO
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
This MR should only be merged after this one - https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/69/diffs
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M1 - Release 0.1Vibhuti Sharma [Microsoft]Vibhuti Sharma [Microsoft]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/127Merge IBM changes into Master2023-08-18T22:25:13ZBhushan RadeMerge IBM changes into MasterPull latest changes for IBM and merge into master
skipped one integration test for IBM -
```
should_createSchema_and_returnHttp409IfTryToCreateItAgain_and_getSchema_and_deleteSchema_when_providingValidSchemaInfo(org.opengroup.osdu.stor...Pull latest changes for IBM and merge into master
skipped one integration test for IBM -
```
should_createSchema_and_returnHttp409IfTryToCreateItAgain_and_getSchema_and_deleteSchema_when_providingValidSchemaInfo(org.opengroup.osdu.storage.schema.TestCreateSchemaIntegration)
```M1 - Release 0.1Anuj GuptaAnuj Guptahttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/128Using BlobStore for all blob related operations2023-08-18T22:25:12ZAbhishek PatilUsing BlobStore for all blob related operations## Type of change
- [ ] Bug Fix
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provi...## Type of change
- [ ] Bug Fix
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Client classes from azure blob storage sdk were being used directly for all blob related operations.
## What is the new/expected behavior?
BlobStore present in common lib is now used to handle all blob related operations.
## Have you added/updated Unit Tests and Integration Tests?
NA. All the existing tests are passing.
## Any other useful informationM1 - Release 0.1Abhishek PatilAbhishek Patilhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/129extend path api with tags support, refactoring;2023-08-18T22:25:10ZYauheni Lesnikauextend path api with tags support, refactoring;Decided to add tag api support through the extension of patch api with ne path - "/tags". It should support 3 types of operations:
- add
- replace
- remove
Also I refactored batch service and added integration tests for the tag function...Decided to add tag api support through the extension of patch api with ne path - "/tags". It should support 3 types of operations:
- add
- replace
- remove
Also I refactored batch service and added integration tests for the tag functionality
Issue: #39
Depends on os-core-common MR: [48](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/48)M4 - Release 0.7ethiraj krishnamanaiduNitin-slbNeelesh ThakurAlok JoshiYauheni Lesnikauethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/130fix null pointer exception when ancestry is empty2023-08-18T22:25:08ZSherman Yangfix null pointer exception when ancestry is emptyThis MR fixes a subtle null pointer exception which could occur in the follow line: !record.getAncestry().getParents().isEmpty()) when the ancestry has no parent. The getParents() would return null. The next call isEmpty() would cause a ...This MR fixes a subtle null pointer exception which could occur in the follow line: !record.getAncestry().getParents().isEmpty()) when the ancestry has no parent. The getParents() would return null. The next call isEmpty() would cause a null pointer exception when called on the returned null parents object.
The fix is to replace the line with the following line: !Collections.isEmpty(record.getAncestry().getParents())) Now if the getParents() method returns null, the Collections.isEmpty(null) call will correctly return true instead of null pointer exception as expected.
Corresponding unit tests have been updated accordingly.M5 - Release 0.8https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/131Use R3 record validation rules2023-08-18T22:25:06ZMatt WiseUse R3 record validation rules# Merge request template
Resolves #44
Logic based on https://community.opengroup.org/osdu/platform/system/storage/-/issues/26# Merge request template
Resolves #44
Logic based on https://community.opengroup.org/osdu/platform/system/storage/-/issues/26M3 - Release 0.5ethiraj krishnamanaiduDania Kodeih (Microsoft)Wladmir FrazaoJoeDmitriy RudkoMatt Wiseethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/134update validation message for invalid record id2023-08-18T22:25:05ZMatt Wiseupdate validation message for invalid record id# Merge request template# Merge request templateM3 - Release 0.5ethiraj krishnamanaiduJoeJasonethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/138Removing synchronized keyword2023-08-18T22:25:03ZAbhishek PatilRemoving synchronized keyword## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provi...## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
The method `get()` of RecordMetadataRepository class was marked synchronized due to which performance of GET /records/{id} and PUT /records api was affected.
## What is the new/expected behavior?
There is no need for the method to be synchronized and hence after removing synchronized keyword the performance of the api's is improved.
## Have you added/updated Unit Tests and Integration Tests?
NA. All the existing ITs are passing.
## Any other useful information
NA.M4 - Release 0.7Abhishek PatilAbhishek Patilhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/139Logging Enhancements for GCP modules (GONRG-1654, GONRG-1766)2023-08-18T22:25:02ZArtem Dobrynin (EPAM)Logging Enhancements for GCP modules (GONRG-1654, GONRG-1766)## Type of change
- [ ] Bug Fix
- [X] Feature
Some enhancements for GCP logging system. Supplies the logs with user id, data-partition-id and correlation id
https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/...## Type of change
- [ ] Bug Fix
- [X] Feature
Some enhancements for GCP logging system. Supplies the logs with user id, data-partition-id and correlation id
https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/-/issues/50 https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/-/issues/49
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
The logs are printed in an unparseable inline format.
## What is the new/expected behavior?
The logs are printed in JSON format and compatible with GCP Logger.
## Have you added/updated Unit Tests and Integration Tests?
No, since it is not required for these changes
## Any other useful information
There are two types of logging: for local Spring profile (`spring.profiles.active=local`) and for any other, incl. default. This was done for better readability while testing locally.
The MR should be merged after https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/os-core-lib-gcp/-/merge_requests/10M4 - Release 0.7Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/140CORS Fix, AWS Updates2023-08-18T22:25:00ZSpencer Suttonsuttonsp@amazon.comCORS Fix, AWS Updates# Merge request template# Merge request templateM4 - Release 0.7ethiraj krishnamanaiduWladmir FrazaoJoeDmitriy Rudkoethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/141Bug: fixing missing CPU limit character.2023-08-18T22:24:58ZHema Vishnu Pola [Microsoft]Bug: fixing missing CPU limit character.## Type of change
- [X] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [YES/NO] NO
## Does this introduce a change in the clou...## Type of change
- [X] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [YES/NO] NO
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [X] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [YES/NO] NO
## What is the current behavior?
Current behavior consume 800 cores of CPU, which is equivalent to infinite.
## What is the new/expected behavior?
This is limited to use 800 millicores, instead of earlier 800 cores.
## Have you added/updated Unit Tests and Integration Tests?
## Any other useful informationM4 - Release 0.7Kishore BattulaKrishna Nikhil VedurumudiKishore Battulahttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/142Adding Cache for TenantInfoDoc2023-08-18T22:24:56ZAbhishek PatilAdding Cache for TenantInfoDoc## Type of change
- [ ] Bug Fix
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provi...## Type of change
- [ ] Bug Fix
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
While processing every `PUT /records` request, we query the TenantInfoDoc based on partition id.
## What is the new/expected behavior?
Caching TenantInfoDoc.
## Have you added/updated Unit Tests and Integration Tests?
NA
## Any other useful informationM4 - Release 0.7Abhishek PatilAbhishek Patil