Storage merge requestshttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests2023-07-07T15:01:26Zhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/724Enhance crs error messages2023-07-07T15:01:26ZAlok JoshiEnhance crs error messagesCapture CRS conversion timeout issue separately from other conversion errors.Capture CRS conversion timeout issue separately from other conversion errors.M19 - Release 0.22Alok JoshiChad LeongAlok Joshihttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/150Event grid topic dynamic addition reaction change2023-08-18T22:24:46ZKomal MakkarEvent grid topic dynamic addition reaction change## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] Does the MR contain pipeline/ helm chart related changes?
* [NA] I have ...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] Does the MR contain pipeline/ helm chart related changes?
* [NA] I have updated the documentation accordingly.
* [NA] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [YES] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
**Original Behavior** The event grid topic was a fixed list.
**Modified behavior**
Event Grid Topics are now getting in the system dynamically. This change is accommodating the same.
High level design:
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO] ~~**YES**~~
~~Waiting for MR https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/107~~
NO. MR merged
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M5 - Release 0.8Kishore BattulaKishore Battulahttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/110Event Grid uses the Abstraction in Core Lib Azure2023-08-18T22:25:33ZKomal MakkarEvent Grid uses the Abstraction in Core Lib Azure## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NO] Does the MR contain pipeline/ helm chart related changes?
* [NO] I have u...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NO] Does the MR contain pipeline/ helm chart related changes?
* [NO] I have updated the documentation accordingly.
* [NO] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [YES] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
MR to be merged after https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/59/diffs
The changes absorb the abstraction of EG topic laid down in CoreLibAzure
High level design:
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO]
Yes. Details in the readme.
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M1 - Release 0.1Kishore BattulaKishore Battulahttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/308Experimental FOSSA caching2023-08-18T22:22:27ZDavid Diederichd.diederich@opengroup.orgExperimental FOSSA cachingThis MR adds in some experimental FOSSA logic, which is intended to ease the pain of using FOSSA. I'd like to add it to one service to get some "real-world" testing, then apply it globally once we see it works reasonably. I'm very intere...This MR adds in some experimental FOSSA logic, which is intended to ease the pain of using FOSSA. I'd like to add it to one service to get some "real-world" testing, then apply it globally once we see it works reasonably. I'm very interested in feedback on how this is working and ways that it could be improved before we make it standard.
Note that this uses the image `fossa-with-cache/incremental:latest` -- meaning it grabs the latest build rather than a stable release one. This is to make it easier to iterate and refine during this testing phase. Before it is moved to the ci-cd-pipelines project, it will be changed to follow a release tag.
### Caching of the NOTICE Files
The primary feature of the [`fossa-with-cache`](https://community.opengroup.org/divido/fossa-with-cache) tool is that it caches the NOTICE that FOSSA generates and associates it to a list of project dependencies. If the dependencies do not change, the tool will re-use the previously generated NOTICE rather than asking FOSSA to create a new one. This should cut down on instances of small, jittery changes in the NOTICE from one generation to the next.
The cache ages out after a period of time (default = 1 week), so the first pipeline run after that age will regenerate even if there's a cached version available. This is to avoid going too long before reconsulting FOSSA -- FOSSA's database is constantly evolving, and it may have new information that the previous run didn't.
### Single package names, list of URLs
When multiple different package names refer to the same package (like spring-security & Spring Security), a single canonical name is used instead. When multiple URLs are used, they are concatenated into a comma separated list. The configuration for this is hard-coded, and will need to be updated over time. But, at least for the known cases, this should cut down on NOTICE differences that are only these typographical changes.
This does mean that every NOTICE file will change the first time the pipeline uses this logic. This is unavoidable, but should only happen once on each project.
### NOTICE files can be updated without re-running the pipeline
If there are new commits to a branch, and the only changes are the NOTICE file, the `fossa-check-notice` stage will compare the generated (or cached) NOTICE against the latest committed version, rather than the one associated with the pipeline. This allows developers to fix a NOTICE file without having to re-run the entire pipeline.
This is a bit counterintuitive from a CI perspective. Normally, you expect that pipelines only operate on the specific commit that generated them. However, in this case, I think the non-standard technique is useful to reduce the cost associated with a NOTICE difference.
The skipped pipeline also shows up in the list of pipelines, including the MR list. We'll need to know to look for previous pipelines if we see a skipped pipeline with commit message "Update NOTICE" or similar.
In the case of trusted branches, the NOTICE should be updated on the regular development branch first. Then, move the trusted branch to match (`git branch -f trusted-devBranch devBranch` or similar) and push it up. Trusted branches already suppress pipelines, so there's no need to push with `-o ci.skip` on that one. Finally, re-run the `fossa-check-notice` on the protected pipeline. If you only update the trusted branch, the pipeline will pass but the NOTICE files won't get merged in. If you only update the development branch, the pipeline won't see them and will continue to fail. I may add some additional commentary to the pipeline failure message to help with this, or perhaps add special logic around the trusted branches to make this easier. Interested in feedback / ideas on this.
**Example: This MR**
For [pipeline 79827](https://community.opengroup.org/osdu/platform/system/storage/-/pipelines/79827), the [original `fossa-check-notice` job (747962)](https://community.opengroup.org/osdu/platform/system/storage/-/jobs/747962) failed with differences in the NOTICE. I followed the steps to download the cached version, and [re-ran the job (748097)](https://community.opengroup.org/osdu/platform/system/storage/-/jobs/748097). The [pipeline (79837)](https://community.opengroup.org/osdu/platform/system/storage/-/pipelines/79837) from the changed NOTICE was skipped; and that shows up in this MR as the latest pipeline.M10 - Release 0.13David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/355Explicit json-smart dependency2022-09-16T02:12:40ZAlok JoshiExplicit json-smart dependencyUpgrading json-smart to 2.4.7Upgrading json-smart to 2.4.7M10 - Release 0.13Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/639Extend patch API to support data block modification2023-09-08T19:26:41ZAlok JoshiExtend patch API to support data block modificationThis change is an effort to implement [ADR](https://community.opengroup.org/osdu/platform/system/storage/-/issues/124).
Some considerations:
- This feature is implemented as a new API. To comply with the [RFC spec](https://www.rfc-edit...This change is an effort to implement [ADR](https://community.opengroup.org/osdu/platform/system/storage/-/issues/124).
Some considerations:
- This feature is implemented as a new API. To comply with the [RFC spec](https://www.rfc-editor.org/rfc/rfc6902.html), API media type must be "application/json-patch+json". The new API uses the same request mapping and HTTP method as the existing bulk update API
- Support for updating data block (i.e. 'data' and 'meta' fields in Record) is added. We are also extending metadata update support for additional fields (ancestry/parents, kind)
- There is no update to storage_openapi.yaml file under /docs/api because the swagger is generated with code now.
Active work items:
- Update tutorial documentation
Note to reviewers:
- There are a lot of seemingly unrelated small unit test changes. We made those because of a change in the mockito dependency, which performs strict mock stubbing checks by default. So, we've relaxed those checks by default in the unit test classes. The mockito dependency change was made because we wanted to introduce WebMvcTest with SpringRunner for the new API. This helps us mock the http call to the new endpointM17 - Release 0.20Alok JoshiChad LeongAlok Joshihttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/129extend path api with tags support, refactoring;2023-08-18T22:25:10ZYauheni Lesnikauextend path api with tags support, refactoring;Decided to add tag api support through the extension of patch api with ne path - "/tags". It should support 3 types of operations:
- add
- replace
- remove
Also I refactored batch service and added integration tests for the tag function...Decided to add tag api support through the extension of patch api with ne path - "/tags". It should support 3 types of operations:
- add
- replace
- remove
Also I refactored batch service and added integration tests for the tag functionality
Issue: #39
Depends on os-core-common MR: [48](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/48)M4 - Release 0.7ethiraj krishnamanaiduNitin-slbNeelesh ThakurAlok JoshiYauheni Lesnikauethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/210Failing tests fix2023-08-18T22:23:38ZHema Vishnu Pola [Microsoft]Failing tests fix## Type of change
- [X] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [YES/NO]
## Does this introduce a change in the cloud p...## Type of change
- [X] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [YES/NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [YES/NO]
## What is the current behavior?
Core common function signature changed, which needs to be taken into the service changes.
## What is the new/expected behavior?
## Have you added/updated Unit Tests and Integration Tests?
## Any other useful information
Link to the changed core-common MR: https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/86M7 - Release 0.10https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/45Feat/aws integration2023-08-18T22:29:29ZSpencer Suttonsuttonsp@amazon.comFeat/aws integration# Merge request template# Merge request templateM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/653Filter metadata records by partition id2023-03-17T16:38:34ZMorris EstepaFilter metadata records by partition id## Type of change
- [X] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
https://community.opengroup.org/osdu/platform/system/storage/-/issues/167
## Does this introduce a change in ...## Type of change
- [X] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
https://community.opengroup.org/osdu/platform/system/storage/-/issues/167
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Records from other partitions are returned when querying the service
## What is the new/expected behavior?
Only records from the target data partition are returned.
## Have you added/updated Unit Tests and Integration Tests?
Yes
## Any other useful informationM16 - Release 0.19Morris EstepaMorris Estepahttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/591fix acl checking on azure CloudStorageImpl for reading version2023-01-12T23:52:48ZYauheni Lesnikaufix acl checking on azure CloudStorageImpl for reading versionISSUE: https://community.opengroup.org/osdu/platform/system/storage/-/issues/158ISSUE: https://community.opengroup.org/osdu/platform/system/storage/-/issues/158M16 - Release 0.19Yauheni LesnikauYauheni Lesnikauhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/739Fix acl duplicates2023-07-28T14:06:27ZAlok JoshiFix acl duplicatesPlease see [this issue](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/71)
Related core-common [MR](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/222...Please see [this issue](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/71)
Related core-common [MR](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/222)
Added an integration testM20 - Release 0.23Alok JoshiChad LeongAlok Joshihttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/797fix aws versions2023-12-05T20:49:17ZYunhua Koglinfix aws versions# Merge request template# Merge request templateM22 - Release 0.25Yunhua KoglinYunhua Koglinhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/859fix azure deploy2024-03-19T12:09:13ZVidyaDharani Lokamfix azure deploy- upgrade dependencies to make compatible with spring6
- fix azure deploy and Integration test failures.- upgrade dependencies to make compatible with spring6
- fix azure deploy and Integration test failures.M23 - Release 0.26VidyaDharani LokamVidyaDharani Lokamhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/61fix bootstrap issues2023-08-18T22:29:12ZAliaksei Darafeyeufix bootstrap issues# Merge request template# Merge request templateM1 - Release 0.1ethiraj krishnamanaiduSherman Yangethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/513fix CrsConversionService potential class cast issues2022-10-02T04:42:54ZYauheni Lesnikaufix CrsConversionService potential class cast issuesIssue: https://community.opengroup.org/osdu/platform/system/storage/-/issues/144Issue: https://community.opengroup.org/osdu/platform/system/storage/-/issues/144M14 - Release 0.17Yauheni LesnikauYauheni Lesnikauhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/451Fix data inconsistency issue between metadata storage and blob storage2022-09-15T10:04:42ZAlok JoshiFix data inconsistency issue between metadata storage and blob storage## Issue
We've observed data inconsistency issues for some storage record objects where the latest version of record (available in metadata storage - i.e. cosmosDb) is soft deleted from blob storage (i.e. Azure Storage). This is causing ...## Issue
We've observed data inconsistency issues for some storage record objects where the latest version of record (available in metadata storage - i.e. cosmosDb) is soft deleted from blob storage (i.e. Azure Storage). This is causing exception for APIs like GET `/api/storage/v2/records/<id>` and POST `/api/storage/v2/query/records:batch`
## Solution
We propose to fix this data inconsistency by two changes:
- Clean up datastore as well, when cleaning up cloud storage, in case of commitBatch method exception in `PersistenceServiceImpl` (core logic change)
- For existing records having this issue: try restoring the data from blob storage. We are attempting to restore the latest record version (recorded in cosmosDb) from blob storage if its not foundM14 - Release 0.17Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/786fix duplicate entries while logging2023-11-13T09:21:53ZVidyaDharani Lokamfix duplicate entries while loggingRelated Issue: https://community.opengroup.org/osdu/platform/system/storage/-/issues/182
As mentioned in the above issue we have same operation_id for disparate operations(based on record ids).
- In this MR enabled sampling for applicat...Related Issue: https://community.opengroup.org/osdu/platform/system/storage/-/issues/182
As mentioned in the above issue we have same operation_id for disparate operations(based on record ids).
- In this MR enabled sampling for application insights to reduce the duplicacy in operation id.
- Logs of QueryApi/getRecords API
![image](/uploads/d71f759d8718d0789bfdf437e9ba2d29/image.png)
- Logs of QueryApi/fetchRecords API
![image](/uploads/a2539327cc09043b8e3e3bfde373542a/image.png)M22 - Release 0.25VidyaDharani LokamVidyaDharani Lokamhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/5Fixed a simple typo that can from updating the document to be more OSDU aligned2023-08-18T22:30:01ZStephen Whitley (Invited Expert)Fixed a simple typo that can from updating the document to be more OSDU aligned• What is the change?
Typo in Swagger description that came from making the text more OSDU friendly
• What does it impact?
• How do I reproduce or test the change?• What is the change?
Typo in Swagger description that came from making the text more OSDU friendly
• What does it impact?
• How do I reproduce or test the change?Release 2.0https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/72Fixed KV env variable2023-08-18T22:28:52ZDaniel SchollFixed KV env variableSimple Env Name ChangeSimple Env Name ChangeM1 - Release 0.1Daniel SchollDaniel Scholl