Storage merge requestshttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests2023-08-18T22:25:20Zhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/121adding MongoDB support for AWS2023-08-18T22:25:20ZAleh Shubko [EPAM]adding MongoDB support for AWSAdded initial MongoDB implementation for AWSAdded initial MongoDB implementation for AWSM3 - Release 0.5Aleh Shubko [EPAM]Aleh Shubko [EPAM]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/123Cloud-agnostic module `storage-reference` for Anthos (GONRG-1423)2023-08-18T22:25:18ZRiabokon Stanislav(EPAM)[GCP]Cloud-agnostic module `storage-reference` for Anthos (GONRG-1423)# Description:
Added a new cloud-agnostic module `storage-reference`. This module can be used for Anthos GCP.
Database: MongoDB
Messaging Broker: RabbitMQ
# How to test:
Bussines cases.
# Changes include:
- [ ] Refactor (a non-break...# Description:
Added a new cloud-agnostic module `storage-reference`. This module can be used for Anthos GCP.
Database: MongoDB
Messaging Broker: RabbitMQ
# How to test:
Bussines cases.
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [X] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
# Dev Checklist:
- [x] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M1 - Release 0.1Dmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/124Azure adding cosmos bulk operations2023-08-18T22:25:16ZJasonAzure adding cosmos bulk operations## Type of change
- [ ] Bug Fix
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
N/A
## Does this introduce a change in the core logic?
- [YES/**NO**]
## Does this introduce a change in the ...## Type of change
- [ ] Bug Fix
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
N/A
## Does this introduce a change in the core logic?
- [YES/**NO**]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [YES/**NO**]
## What is the current behavior?
- For any batch size, each record is queried in Cosmos individually to see if it exists
- For any batch size, the records are uploaded one at a time to Cosmos
- Old unused pipelines are in the service repo
## What is the new/expected behavior?
- An environment variable determines the minimum batch size to use the DocumentBulkExecutor to upload the records in parallel. If the batch size is >= this minimum, the records will be uploaded in parallel using bulk executor. If the batch size is less than the minimum, the records are uploaded in serial as they were before.
- For all batch sizes, records are queried using a SQL query of the form `SELECT * FROM c WHERE c.id IN [list_of_ids_here]` to reduce time to query for large batch sizes.
- Old pipelines are deleted
## Have you added/updated Unit Tests and Integration Tests?
## Any other useful information
- MR adding bulk executor to core-lib-azure can be found [here](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/66).M1 - Release 0.1JasonJasonhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/125add label property to the message of the azure event grid message, for...2021-04-05T09:53:47ZYauheni Lesnikauadd label property to the message of the azure event grid message, for...The amendment adds the label property to the event grid message (Azure implementation of IMessageBus) for the further filtering messages into the Azure Service Bus Subscribtions.
Issue: https://community.opengroup.org/osdu/platform/syste...The amendment adds the label property to the event grid message (Azure implementation of IMessageBus) for the further filtering messages into the Azure Service Bus Subscribtions.
Issue: https://community.opengroup.org/osdu/platform/system/storage/-/issues/43ethiraj krishnamanaiduDania Kodeih (Microsoft)Nitin-slbNeelesh Thakurashley kelhamDuvelis CaraoYauheni Lesnikauethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/126Adding CustomThreadPoolFactory2023-08-18T22:25:15ZVibhuti Sharma [Microsoft]Adding CustomThreadPoolFactory## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NO] Does the MR contain pipeline/ helm chart related changes?
* [NO] I have u...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NO] Does the MR contain pipeline/ helm chart related changes?
* [NO] I have updated the documentation accordingly.
* [NA] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [NA] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
High level design:
Adding CustomThreadPoolFactory to enable creation of `CustomThreadPoolExecutor` from core-lib-azure, which allows for copying of context from main thread to child threads.
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
Concurrency classes being used from the java.util.concurrency package do not allow copying of MDC and request context from main thread to child threads.
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
## Does this introduce a breaking change?
-------------------------------------
- NO
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
This MR should only be merged after this one - https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/69/diffs
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M1 - Release 0.1Vibhuti Sharma [Microsoft]Vibhuti Sharma [Microsoft]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/127Merge IBM changes into Master2023-08-18T22:25:13ZBhushan RadeMerge IBM changes into MasterPull latest changes for IBM and merge into master
skipped one integration test for IBM -
```
should_createSchema_and_returnHttp409IfTryToCreateItAgain_and_getSchema_and_deleteSchema_when_providingValidSchemaInfo(org.opengroup.osdu.stor...Pull latest changes for IBM and merge into master
skipped one integration test for IBM -
```
should_createSchema_and_returnHttp409IfTryToCreateItAgain_and_getSchema_and_deleteSchema_when_providingValidSchemaInfo(org.opengroup.osdu.storage.schema.TestCreateSchemaIntegration)
```M1 - Release 0.1Anuj GuptaAnuj Guptahttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/128Using BlobStore for all blob related operations2023-08-18T22:25:12ZAbhishek PatilUsing BlobStore for all blob related operations## Type of change
- [ ] Bug Fix
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provi...## Type of change
- [ ] Bug Fix
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Client classes from azure blob storage sdk were being used directly for all blob related operations.
## What is the new/expected behavior?
BlobStore present in common lib is now used to handle all blob related operations.
## Have you added/updated Unit Tests and Integration Tests?
NA. All the existing tests are passing.
## Any other useful informationM1 - Release 0.1Abhishek PatilAbhishek Patilhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/129extend path api with tags support, refactoring;2023-08-18T22:25:10ZYauheni Lesnikauextend path api with tags support, refactoring;Decided to add tag api support through the extension of patch api with ne path - "/tags". It should support 3 types of operations:
- add
- replace
- remove
Also I refactored batch service and added integration tests for the tag function...Decided to add tag api support through the extension of patch api with ne path - "/tags". It should support 3 types of operations:
- add
- replace
- remove
Also I refactored batch service and added integration tests for the tag functionality
Issue: #39
Depends on os-core-common MR: [48](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/48)M4 - Release 0.7ethiraj krishnamanaiduNitin-slbNeelesh ThakurAlok JoshiYauheni Lesnikauethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/130fix null pointer exception when ancestry is empty2023-08-18T22:25:08ZSherman Yangfix null pointer exception when ancestry is emptyThis MR fixes a subtle null pointer exception which could occur in the follow line: !record.getAncestry().getParents().isEmpty()) when the ancestry has no parent. The getParents() would return null. The next call isEmpty() would cause a ...This MR fixes a subtle null pointer exception which could occur in the follow line: !record.getAncestry().getParents().isEmpty()) when the ancestry has no parent. The getParents() would return null. The next call isEmpty() would cause a null pointer exception when called on the returned null parents object.
The fix is to replace the line with the following line: !Collections.isEmpty(record.getAncestry().getParents())) Now if the getParents() method returns null, the Collections.isEmpty(null) call will correctly return true instead of null pointer exception as expected.
Corresponding unit tests have been updated accordingly.M5 - Release 0.8https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/131Use R3 record validation rules2023-08-18T22:25:06ZMatt WiseUse R3 record validation rules# Merge request template
Resolves #44
Logic based on https://community.opengroup.org/osdu/platform/system/storage/-/issues/26# Merge request template
Resolves #44
Logic based on https://community.opengroup.org/osdu/platform/system/storage/-/issues/26M3 - Release 0.5ethiraj krishnamanaiduDania Kodeih (Microsoft)Wladmir FrazaoJoeDmitriy RudkoMatt Wiseethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/132CORS Fix2021-03-05T21:22:52ZSpencer Suttonsuttonsp@amazon.comCORS Fix# Merge request template# Merge request templateSpencer Suttonsuttonsp@amazon.comSpencer Suttonsuttonsp@amazon.comhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/133Azure: Updating os-core-common version2021-01-29T03:10:55ZJasonAzure: Updating os-core-common versionAzure currently fails test step on this branch because we explicitly define an older version of os-core-common in our POM. This change updates that to the new required version.Azure currently fails test step on this branch because we explicitly define an older version of os-core-common in our POM. This change updates that to the new required version.JasonJasonhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/134update validation message for invalid record id2023-08-18T22:25:05ZMatt Wiseupdate validation message for invalid record id# Merge request template# Merge request templateM3 - Release 0.5ethiraj krishnamanaiduJoeJasonethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/135update response header addition sequence as spring does not allow this once r...2021-03-05T16:35:20ZNeelesh Thakurupdate response header addition sequence as spring does not allow this once response is committed# Merge request template# Merge request templatehttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/136Fail fast before fetching blob2021-02-09T22:33:13ZHema Vishnu Pola [Microsoft]Fail fast before fetching blob## Type of change
- [X] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [YES] - Instead of waiting to fetch blob and then failin...## Type of change
- [X] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [YES] - Instead of waiting to fetch blob and then failing in ACL, made to fail fast and fetch the blob post that. Please do validate if it does make sense and doesn't disturb any of the flow.
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
After fetching the blob, then ACL checks are done.
## What is the new/expected behavior?
Reversing the order to fetch blob only when ACL checks succeed. This would allow to fail fast before loading the whole blob.
## Have you added/updated Unit Tests and Integration Tests?
## Any other useful informationhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/137WIP: (GONRG-1838) GCP cannot retrieve a record with space2021-03-03T02:48:14ZIgor Filippov (EPAM)WIP: (GONRG-1838) GCP cannot retrieve a record with spaceThere is a **core** issue with record IDs that contain encrypted space symbols (%20):
**Create record RQ:**
```
curl --location --request PUT 'https://os-storage-attcrcktoa-uc.a.run.app/api/storage/v2/records' \
--header 'Data-Partition...There is a **core** issue with record IDs that contain encrypted space symbols (%20):
**Create record RQ:**
```
curl --location --request PUT 'https://os-storage-attcrcktoa-uc.a.run.app/api/storage/v2/records' \
--header 'Data-Partition-ID: osdu' \
--header 'Authorization: Bearer <token>' \
--header 'Content-Type: application/json' \
--data-raw '[{
"id": "osdu:work-product-component--WellboreMarkerSet:Some%20Text",
...
}]'
```
**Get record RQ:**
```
curl --location --request GET 'https://os-storage-attcrcktoa-uc.a.run.app/api/storage/v2/records/osdu:work-product-component--WellboreMarkerSet:Some%20Text' \
--header 'Data-Partition-ID: osdu' \
--header 'Authorization: Bearer <token>'
```
**Get record RS:**
```
{
"code": 400,
"reason": "Validation error.",
"message": "getLatestRecordVersion.id: Not a valid record id. Found: osdu:work-product-component--WellboreMarkerSet:Some Text"
}
```
**Expected Result**
1. I should be able to retrieve it by ID that was used during record creation
**Actual Result**
2. Its not possible to retrieve the record after
## Type of change
- [X] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
- [x] Core
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
It is impossible to retrieve record from the Storage service
## What is the new/expected behavior?
It is possible to retrieve record from the Storage service
## Have you added/updated Unit Tests and Integration Tests?
- [NO]Dmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/138Removing synchronized keyword2023-08-18T22:25:03ZAbhishek PatilRemoving synchronized keyword## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provi...## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
The method `get()` of RecordMetadataRepository class was marked synchronized due to which performance of GET /records/{id} and PUT /records api was affected.
## What is the new/expected behavior?
There is no need for the method to be synchronized and hence after removing synchronized keyword the performance of the api's is improved.
## Have you added/updated Unit Tests and Integration Tests?
NA. All the existing ITs are passing.
## Any other useful information
NA.M4 - Release 0.7Abhishek PatilAbhishek Patilhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/139Logging Enhancements for GCP modules (GONRG-1654, GONRG-1766)2023-08-18T22:25:02ZArtem Dobrynin (EPAM)Logging Enhancements for GCP modules (GONRG-1654, GONRG-1766)## Type of change
- [ ] Bug Fix
- [X] Feature
Some enhancements for GCP logging system. Supplies the logs with user id, data-partition-id and correlation id
https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/...## Type of change
- [ ] Bug Fix
- [X] Feature
Some enhancements for GCP logging system. Supplies the logs with user id, data-partition-id and correlation id
https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/-/issues/50 https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/-/issues/49
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
The logs are printed in an unparseable inline format.
## What is the new/expected behavior?
The logs are printed in JSON format and compatible with GCP Logger.
## Have you added/updated Unit Tests and Integration Tests?
No, since it is not required for these changes
## Any other useful information
There are two types of logging: for local Spring profile (`spring.profiles.active=local`) and for any other, incl. default. This was done for better readability while testing locally.
The MR should be merged after https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/os-core-lib-gcp/-/merge_requests/10M4 - Release 0.7Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/140CORS Fix, AWS Updates2023-08-18T22:25:00ZSpencer Suttonsuttonsp@amazon.comCORS Fix, AWS Updates# Merge request template# Merge request templateM4 - Release 0.7ethiraj krishnamanaiduWladmir FrazaoJoeDmitriy Rudkoethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/141Bug: fixing missing CPU limit character.2023-08-18T22:24:58ZHema Vishnu Pola [Microsoft]Bug: fixing missing CPU limit character.## Type of change
- [X] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [YES/NO] NO
## Does this introduce a change in the clou...## Type of change
- [X] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [YES/NO] NO
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [X] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [YES/NO] NO
## What is the current behavior?
Current behavior consume 800 cores of CPU, which is equivalent to infinite.
## What is the new/expected behavior?
This is limited to use 800 millicores, instead of earlier 800 cores.
## Have you added/updated Unit Tests and Integration Tests?
## Any other useful informationM4 - Release 0.7Kishore BattulaKrishna Nikhil VedurumudiKishore Battula