Storage merge requestshttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests2023-08-18T22:29:27Zhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/47Revert "Merge branch 'feat/aws-integration' into 'master'"2023-08-18T22:29:27ZSpencer Suttonsuttonsp@amazon.comRevert "Merge branch 'feat/aws-integration' into 'master'"# Merge request template# Merge request templateM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/45Feat/aws integration2023-08-18T22:29:29ZSpencer Suttonsuttonsp@amazon.comFeat/aws integration# Merge request template# Merge request templateM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/44Bulk update api & GCP implementation2023-08-18T22:29:31ZHong YanBulk update api & GCP implementation## Type of change
- [ ] Bug Fix
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
[[System/Storage] Bulk Update for ACLs on Records](https://community.opengroup.org/osdu/platform/system/storag...## Type of change
- [ ] Bug Fix
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
[[System/Storage] Bulk Update for ACLs on Records](https://community.opengroup.org/osdu/platform/system/storage/-/issues/10)
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x] IBM
The API is fully implemented for GCP. In storage core, the new added interface methods
```java
org.opengroup.osdu.storage.provider.interfaces.ICloudStorage.updateObjectMetadata
org.opengroup.osdu.storage.provider.interfaces.ICloudStorage.revertObjectMetadata
```
A new method is added in API and two methods added in SPI. We have overridden the method in all clouds which throws NotImplementedException, this allows us to review and merge the changes now and then cloud providers can implement depends on the availability.
## Does this introduce a breaking change?
- [YES]
## What is the current behavior?
Records that are ingested with ACLs that are incorrect can be painful to correct in the current system. There is no update mechanism for the ACLs, leading to mass re-ingestions of records for the sole purpose of correcting ACLs when the data has not changed.
## What is the new/expected behavior?
Add a new API (PATCH /records)to do PATCH updates of just the ACLs of a batch of data so that the only input needs to be the Record Identifiers and the new ACLs.
## Have you added/updated Unit Tests and Integration Tests?
Yes, new unit tests are added for storage-core logic and gcp implementation. Integration tests are also added, but need to be tested in a trusted branch to ensure they work properly. For all providers other than GCP, the integration tests are overridden by placeholder tests which only check if the api returns 501 Not Implemented.
## Any other useful informationM1 - Release 0.1ethiraj krishnamanaiduDania Kodeih (Microsoft)JoeBrandt BealDaniel SchollDmitriy RudkoAlan Brazethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/43Merge request template2023-08-18T22:29:32Zethiraj krishnamanaiduMerge request templateAdding the Merge request template.Adding the Merge request template.M1 - Release 0.1https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/42Enforce record owner access for Update/Purge records2023-08-18T22:29:34ZJohnny Guojguo26@slb.comEnforce record owner access for Update/Purge records[Issue 17](https://community.opengroup.org/osdu/platform/system/storage/-/issues/17)
- What is the change?
This change is to make sure that if a user wants to purge/update a record, the user must have access to record ACL groups, which ...[Issue 17](https://community.opengroup.org/osdu/platform/system/storage/-/issues/17)
- What is the change?
This change is to make sure that if a user wants to purge/update a record, the user must have access to record ACL groups, which means be a member of any ACL owner group of the record.
- What does it impact?
If a user does not belong to any ACL group of the record, he/she cannot update/purge it after the PR is merged.
- How do I reproduce or test the change?
Here I add and update unit tests and integration tests, you can also manually test the deployment.
- Other?
I saw some files were using tab indent, so I have changed use 4spaces indent in this PR, so there are files with whole file changes, so please ignore space changes when reviewing PR.
**Please uncheck "show space change" while reviewing code changes, instructions as the following picture:**
![image](/uploads/1e526769662e33e38bd20606d23ef8e0/image.png)M1 - Release 0.1ethiraj krishnamanaiduDania Kodeih (Microsoft)Nitin-slbNeelesh ThakurDaniel SchollAlok JoshiHong Yanethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/40Fix: revert "Adding query all kinds bug fix, no longer filters by user"2023-08-18T22:29:36ZBrandt BealFix: revert "Adding query all kinds bug fix, no longer filters by user"• What is the change? Revert of previous commit
• What does it impact? AWS/Build
• How do I reproduce or test the change? N/A• What is the change? Revert of previous commit
• What does it impact? AWS/Build
• How do I reproduce or test the change? N/AM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/39Adding query all kinds bug fix, no longer filters by user2023-08-18T22:29:37ZBrandt BealAdding query all kinds bug fix, no longer filters by user• What is the change? AWS code had a small problem where query all kinds filtered by user
• What does it impact? Just AWS code
• How do I reproduce or test the change? Tested by integration tests• What is the change? AWS code had a small problem where query all kinds filtered by user
• What does it impact? Just AWS code
• How do I reproduce or test the change? Tested by integration testsM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/35Add UnrecognizedPropertyException and JsonProcessingException handler2023-08-18T22:29:39ZSviatoslav NekhaienkoAdd UnrecognizedPropertyException and JsonProcessingException handler• What is the change?
Handling of UnrecognizedPropertyException and JsonProcessingException have been added
• What does it impact?
BAD_REQUEST is returned instead of INTERNAL_SERVER_ERROR
• How do I reproduce or test the change?
2 Unit...• What is the change?
Handling of UnrecognizedPropertyException and JsonProcessingException have been added
• What does it impact?
BAD_REQUEST is returned instead of INTERNAL_SERVER_ERROR
• How do I reproduce or test the change?
2 Unit tests have been added, BAD_REQUEST is returned in case of wrong Json or property valuesM1 - Release 0.1ethiraj krishnamanaiduNeelesh ThakurSviatoslav Nekhaienkoethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/34fixed storage-test-core maven dependency version2020-07-13T18:03:40Zsatish kumar narganifixed storage-test-core maven dependency version• Fixed the version of storage test core in the storage-test-azure pom.xml• Fixed the version of storage test core in the storage-test-azure pom.xmlM1 - Release 0.1Christian LecknerDania Kodeih (Microsoft)Nandu MuralidharanDaniel SchollChristian Lecknerhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/33Azure - (3/4) - Adding missing context path for Storage2020-07-17T19:08:10ZDuvelis CaraoAzure - (3/4) - Adding missing context path for Storage**This impacts only Azure Implementation**
• What is the change?
Adding context path to Azure Storage: /api/storage/v2/
• What does it impact?
New URL for storage endpoint
• How do I reproduce or test the change?
This MR ...**This impacts only Azure Implementation**
• What is the change?
Adding context path to Azure Storage: /api/storage/v2/
• What does it impact?
New URL for storage endpoint
• How do I reproduce or test the change?
This MR is part of group of MRs with dependencies between them
0. Update infrastructure (Azure / osdu-infrastructure) to have reference to the new URLs (https://github.com/Azure/osdu-infrastructure/pull/23)
1. Adding missing context path for entitlements.
2. Adding missing context path for legal.
3. Adding missing context path for storage.
4. Adding missing context path for indexer
After merging this MR, the new Storage URL will be:
https://{host}**/api/storage/v2/**
Before merging this MR:
- Please make sure the dependent MR is done: https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/merge_requests/32
- Update environment variable (AZURE - Common) with the new storage URL:
STORAGE_URL: https://$ (AZURE_STORAGE_SERVICE_NAME).azurewebsites.net/api/storage/v2/
---------------------
This PR is related to 6 other requests for changing a URL structure and adding in context paths for services.
1. osdu-infrastructure 1/6 PR https://github.com/Azure/osdu-infrastructure/pull/23
1. CI-CD Pipelines 2/6 This PR
1. Entitlements Azure 3/6 https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements-azure/-/merge_requests/11
1. Legal 4/6 https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/merge_requests/32
1. Storage 5/6 https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/33
1. Indexer 6/6 https://community.opengroup.org/osdu/platform/system/indexer-service
They are all required and any one but not the other will break builds and tests across the entire ecosystem for Azure.M1 - Release 0.1ethiraj krishnamanaiduDania Kodeih (Microsoft)Daniel Schollethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/30Searialization bug in QueryItem fixed.2023-08-18T22:29:40ZKishore BattulaSearialization bug in QueryItem fixed.• What is the change?
Exception in searialization when querying kinds is fixed.
• What does it impact?
Querying kinds using /query/kind API
• How do I reproduce or test the change?
On master try to hit the api query/kind API when there...• What is the change?
Exception in searialization when querying kinds is fixed.
• What does it impact?
Querying kinds using /query/kind API
• How do I reproduce or test the change?
On master try to hit the api query/kind API when there are more than 250 kinds in the database.M1 - Release 0.1Daniel SchollImran SiddiqueDaniel Schollhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/24OpenDES GA code merge2023-08-18T22:29:42Zethiraj krishnamanaiduOpenDES GA code merge• What is the change?
* Move on-behalf-of & account-id headers from apis
* Misc fixes on migration code and record update only take changes in CloudStorageImpl.java where it checks !record.hasVersion().
* Separate out NotAllowedExcep...• What is the change?
* Move on-behalf-of & account-id headers from apis
* Misc fixes on migration code and record update only take changes in CloudStorageImpl.java where it checks !record.hasVersion().
* Separate out NotAllowedException in GlobalExceptionMapper
* Change purge to delete
* Add audit logging to records:batch api
* Separate out NotAllowedException in GlobalExceptionMapper
* Change purge to delete
* Add audit logging to records:batch api
* Shorten group cache interval
• What does it impact?
It impacts mostly on the core logic and some cloud providers
• How do I reproduce or test the change?
Integration rest should pass for all cloud providersM1 - Release 0.1ethiraj krishnamanaiduBrandt BealDaniel SchollDmitriy RudkoAlan Brazethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/22Merged PR 1141: Adding missing licenses to files2023-08-18T22:29:44ZAlan BrazMerged PR 1141: Adding missing licenses to files• What is the change?
• What does it impact?
• How do I reproduce or test the change?• What is the change?
• What does it impact?
• How do I reproduce or test the change?M1 - Release 0.1David Diederichd.diederich@opengroup.orgethiraj krishnamanaiduDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/20Google transaction rollback2023-08-18T22:29:45Zethiraj krishnamanaiduGoogle transaction rollback• What is the change?
Only google implementation of code is updated.
• What does it impact?
A couple of tests were failing because of the Google firestorm and the google support team confirmed a slight difference in how a transacti...• What is the change?
Only google implementation of code is updated.
• What does it impact?
A couple of tests were failing because of the Google firestorm and the google support team confirmed a slight difference in how a transaction is handled in Datastore vs Firestore. The solution is to either close or rollback the transaction.
• How do I reproduce or test the change?M1 - Release 0.1ethiraj krishnamanaiduBrandt Bealethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/18AWS variables2023-08-18T22:29:47ZSrihari PrabaharanAWS variablesAWS variables for CI/CDAWS variables for CI/CDM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/16The GCP support team is working on identifying the Firestore issue that is...2023-08-18T22:29:48Zethiraj krishnamanaiduThe GCP support team is working on identifying the Firestore issue that is...• What is the change?
Removed SOP and Thread.Sleep
• What does it impact?
• How do I reproduce or test the change?• What is the change?
Removed SOP and Thread.Sleep
• What does it impact?
• How do I reproduce or test the change?M1 - Release 0.1ethiraj krishnamanaiduBrandt Bealethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/15Azure and GCP pipeline merge to Master :-)2023-08-18T22:29:50Zethiraj krishnamanaiduAzure and GCP pipeline merge to Master :-)• What is the change?
Updated GCP Integration settings and tested in the branch.
• What does it impact?
Deployment config changes
• How do I reproduce or test the change?
The branch pipeline is green.• What is the change?
Updated GCP Integration settings and tested in the branch.
• What does it impact?
Deployment config changes
• How do I reproduce or test the change?
The branch pipeline is green.M1 - Release 0.1ethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/13Pipeline cleanup2023-08-18T22:29:52ZDavid Diederichd.diederich@opengroup.orgPipeline cleanupThis updates the pipeline to use individual includes. I've also fixed a FOSSA error, and suppressed the eslint scanner from running -- GitLab gets tricked into thinking there's javascript in this project because of the presence of an HTM...This updates the pipeline to use individual includes. I've also fixed a FOSSA error, and suppressed the eslint scanner from running -- GitLab gets tricked into thinking there's javascript in this project because of the presence of an HTML file, but there isn't any.M1 - Release 0.1https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/12Enable Azure Cloud Provider Build, Deploy, Test2023-08-18T22:29:54ZDaniel SchollEnable Azure Cloud Provider Build, Deploy, TestThis Merge Request enables Azure.
1. gitlab-ci.yml -- Add Variables and reference the azure template.
2. community-maven-settings.xml -- Add Azure Server for Deploy
3. azure-pipeline.yml -- Moved and initiated an ADO Pipeline to w...This Merge Request enables Azure.
1. gitlab-ci.yml -- Add Variables and reference the azure template.
2. community-maven-settings.xml -- Add Azure Server for Deploy
3. azure-pipeline.yml -- Moved and initiated an ADO Pipeline to work with gitlab
4. Readme -- Azure Provider Readme updatesM1 - Release 0.1Daniel SchollDaniel Schollhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/11Generate required test data in the integration tests2023-08-18T22:29:56ZWyatt NielsenGenerate required test data in the integration testsUpdated integration test setup to generate data for tests instead of depending on data already in environment. Update cloudStorageImpl and RecordsMetadataRepositoryImpl to handle user verification in the same way as other osdu implementa...Updated integration test setup to generate data for tests instead of depending on data already in environment. Update cloudStorageImpl and RecordsMetadataRepositoryImpl to handle user verification in the same way as other osdu implementationsM1 - Release 0.1Brandt BealBrandt Beal