Storage merge requestshttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests2023-08-18T22:22:54Zhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/267Add Virtual Service2023-08-18T22:22:54ZSanjeev-SLBAdd Virtual ServiceIssue [32](https://community.opengroup.org/osdu/platform/system/notification/-/issues/32)Issue [32](https://community.opengroup.org/osdu/platform/system/notification/-/issues/32)M9 - Release 0.12Sanjeev-SLBSanjeev-SLBhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/3ado-codemerge2023-08-18T22:30:04Zethiraj krishnamanaiduado-codemerge• What is the change?
ADO master code merge and removed SLB reference
• What does it impact?
No impact, initial push.
• How do I reproduce or test the change?
Integration test• What is the change?
ADO master code merge and removed SLB reference
• What does it impact?
No impact, initial push.
• How do I reproduce or test the change?
Integration testRelease 2.0https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/539ADR 92: Populate recordBlocks in record modification events2022-12-08T22:56:14ZAbhishek NandaADR 92: Populate recordBlocks in record modification events## Type of change
- [x] Bug Fix
- [X] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
[ADR 92](https://community.opengroup.org/osdu/platform/system/home/-/issues/92)
## Does this introduce a chang...## Type of change
- [x] Bug Fix
- [X] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
[ADR 92](https://community.opengroup.org/osdu/platform/system/home/-/issues/92)
## Does this introduce a change in the core logic?
- [YES/NO] No
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [x] IBM (Added an additional field (hash) to the document db. Auto populated for all other providers.)
## Does this introduce a breaking change?
- [YES/NO] No
## What is the current behavior?
- skipdupes only works for changes in RecordData and ignores changes in RecordMetadata.
- Record modification events do not contain info on which section was updated
## What is the new/expected behavior?
- skipdupes=true now handles for change in any of RecordData and RecordMetadata fields
- This also implements [ADR 92](https://community.opengroup.org/osdu/platform/system/home/-/issues/92) which populates a recordBlocks field with the fields updated
- skipdupes now uses the recordBlock field to check if any updates are made to the record
## Have you added/updated Unit Tests and Integration Tests?
Unit Tests for all new files and new lines with 100% coverage has been added
## Any other useful information
The hashes of data and meta field are now saved in metadata which is used to validate if any changes were made and prevents and extra call to the blob storage for each record.
This is used both in skipdupes=true scenario as well as to populate the recordBlocks field.
This brings around a 50% performance improvement in skipdupes=true scenariosM15 - Release 0.18https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/645Allocating more memory to container than JVM heap memory2023-03-10T18:24:54ZManish JangidAllocating more memory to container than JVM heap memory# Merge request template
Updating container memory# Merge request template
Updating container memoryM16 - Release 0.19https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/839allow id with dot if request with and without dots are in separate requests2024-02-01T22:44:08ZNeelesh Thakurallow id with dot if request with and without dots are in separate requestsEarlier fix on the issue #213 was a breaking change for the consumers. This change-set mitigate the breaking change to certain extent.
OSDU record-ids support trailing dot (.) as defined by Data Definition [schema pattern](https://gitla...Earlier fix on the issue #213 was a breaking change for the consumers. This change-set mitigate the breaking change to certain extent.
OSDU record-ids support trailing dot (.) as defined by Data Definition [schema pattern](https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/blame/master/Generated/abstract/AbstractSystemProperties.1.0.0.json?ref_type=heads#L14). This exact pattern is also used by Storage service's [record-id validator](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blame/1ce74b5eaebdf6b4390c1b50d3ad0e4b76b46e73/src/main/java/org/opengroup/osdu/core/common/model/storage/validation/ValidationDoc.java#L25). As this pattern is defined on reference schema, and used by almost all schemas on OSDU Data Platform, any change to record-id pattern will require major version upgrade to virtually all schemas.
As mentioned on the issue, [Azure Cloud Storage](https://learn.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-containers--blobs--and-metadata) does NOT support trailing dot (.) and hence similar shortcoming are imposed on Storage API.
There are few solutions that can be implemented to address shortcoming in Azure Cloud Storage, but none of those solutions are trivial and require lot of refactoring of common code or low level cloud storage classes on Storage service.
The specific issue only happens when record with and without trailing dot (.) are used on same PUT API call. If they are split in different requests then we can process such record-ids. Moreover, usage of such record-ids are only observed in reference-schema's OSDU bootstrapped data and it's not widely used.
This changeset loosen up previous [validation ](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/825) that outright reject all requests with trailing dot (.). It will still be a breaking change if users have automated workflow to create records that includes record-ids with and without trailing dot (.) on same request, but it will not completely drop the support the record-id with trailing dot (.).M23 - Release 0.26https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/103allow usage of os-core-lib-aws from GL2023-08-18T22:27:04ZRucha Deshpandeallow usage of os-core-lib-aws from GL# Merge request template# Merge request templateM1 - Release 0.1Rucha DeshpandeRucha Deshpandehttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/303apply date and unti conversion for records with AsIngestedCoordinates2022-09-16T08:15:48ZNeelesh Thakurapply date and unti conversion for records with AsIngestedCoordinatesRelated Issue: https://community.opengroup.org/osdu/platform/system/storage/-/issues/99Related Issue: https://community.opengroup.org/osdu/platform/system/storage/-/issues/99M10 - Release 0.13https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/81Audit events for storage (GONRG-941)2023-08-18T22:27:24ZRiabokon Stanislav(EPAM)[GCP]Audit events for storage (GONRG-941)## Type of change
- [ ] Bug Fix
- [X] Feature
https://community.opengroup.org/osdu/platform/deployment-and-operations/audit-and-metrics/-/issues/15
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a c...## Type of change
- [ ] Bug Fix
- [X] Feature
https://community.opengroup.org/osdu/platform/deployment-and-operations/audit-and-metrics/-/issues/15
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [X] Azure
- [X] GCP
- [X] IBM
## Does this introduce a breaking change?
- [YES]
## What is the current behavior?
This event is logged with a hard-coded phrase "All Kinds"
## What is the new/expected behavior?
This event is logged with all kinds.
for example:
`opendes:bulkupdate:test:1.1.1599131561064, opendes:bulkupdate:test:1.1.1599132153171, opendes:bulkupdate:test:1.1.1599132483332, opendes:bulkupdate:test:1.1.1599132676337, opendes:bulkupdate:test:1.1.1599133046031, opendes:bulkupdate:test:1.1.1599133394096, opendes:bulkupdate:test:1.1.1599133437936, opendes:bulkupdate:test:1.1.1599133526479, opendes:bulkupdate:test:1.1.1602591024552, opendes:bulkupdate:test:1.1.1602591910335`
## Have you added/updated Unit Tests and Integration Tests?
- [YES]
## Any other useful informationM1 - Release 0.1Dmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/843auth test cases fix2024-03-19T17:13:56ZNikhil Patilnikhil.patil5@ibm.comauth test cases fix# Merge request template# Merge request templateM23 - Release 0.26Nikhil Patilnikhil.patil5@ibm.comNikhil Patilnikhil.patil5@ibm.comhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/59AWS bulk-acl implementation2020-09-04T16:19:18ZMatt WiseAWS bulk-acl implementation# Merge request template# Merge request templateM1 - Release 0.1Rucha DeshpandeMatt WiseRucha Deshpandehttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/849Aws code quality improvement2024-02-29T16:17:01ZBruce JinAws code quality improvementThis MR will improve AWS code qualityThis MR will improve AWS code qualityM23 - Release 0.26Bruce JinBruce Jinhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/824AWS collaboration header support2024-01-23T14:35:50ZMark ChanceAWS collaboration header support# Merge request template# Merge request templateM23 - Release 0.26https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/51AWS Ecs impl changes2023-08-18T22:29:21ZMatt WiseAWS Ecs impl changes# Merge request template# Merge request templateM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/256AWS Fixes2023-08-18T22:23:06ZMatt WiseAWS Fixes# Merge request template# Merge request templateM8 - Release 0.11Matt WiseMatt Wisehttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/864AWS - Fix internal codebuild build2024-03-25T21:15:48ZGuillaume CailletAWS - Fix internal codebuild buildAWS change only
Fix CodeBuild after spring 6 upgradeAWS change only
Fix CodeBuild after spring 6 upgradeM23 - Release 0.26Guillaume CailletGuillaume Caillethttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/507AWS Helm Build Update2022-09-20T22:12:04ZMarc Burnie [AWS]AWS Helm Build UpdateM14 - Release 0.17Marc Burnie [AWS]Marc Burnie [AWS]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/48Aws integration2023-08-18T22:29:26ZMatt WiseAws integration# Merge request template# Merge request templateM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/678Aws integration merge2023-07-04T11:19:59ZYash DholakiaAws integration mergeAws integration mergeAws integration mergeM18 - Release 0.21Yash DholakiaYash Dholakiahttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/566Aws integration merge2022-12-02T22:15:11ZMichael SaccoAws integration mergeUpdate com.google.oauth-client library to version 1.34.1Update com.google.oauth-client library to version 1.34.1M15 - Release 0.18Michael SaccoMichael Saccohttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/148Aws integration only2023-08-18T22:24:49ZSpencer Suttonsuttonsp@amazon.comAws integration only# Merge request template# Merge request templateM4 - Release 0.7Matt WiseMatt Wise