Storage merge requestshttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests2024-01-03T20:55:00Zhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/812Change Cosmos DB service unavailable error code to 5032024-01-03T20:55:00ZNeha KhandelwalChange Cosmos DB service unavailable error code to 503Currently, Storage batch API responds with 500 error code due to ServiceUnavailableException on CosmosDB. Indexer service, which uses Storage batch API, does not retry if downstream services are responding with 500 error code. Storage ba...Currently, Storage batch API responds with 500 error code due to ServiceUnavailableException on CosmosDB. Indexer service, which uses Storage batch API, does not retry if downstream services are responding with 500 error code. Storage batch API should respond with 503 error code when CosmosDB service cannot be reached to allow Indexer to retry.M23 - Release 0.26https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/809Gonrg 8958 disable policy2023-12-28T13:29:02ZSiarhei Poliak [EPAM / GCP]Gonrg 8958 disable policy# Merge request template# Merge request templateM23 - Release 0.26https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/808Cherry-pick 'Full Upgrade of First Party Library Dependencies for Release 0.2...2023-12-16T10:24:13ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Full Upgrade of First Party Library Dependencies for Release 0.25' into release/0.25**Original MR**: !806
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !806
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/storage/-/pipelines/new?ref=cherry-pick-for-806)M22 - Release 0.25David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/806Full Upgrade of First Party Library Dependencies for Release 0.252023-12-15T20:06:24ZDavid Diederichd.diederich@opengroup.orgFull Upgrade of First Party Library Dependencies for Release 0.25This generated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to try to fully upgrade all dependent libraries to see if the latest code will work.
It is expected that these will ...This generated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to try to fully upgrade all dependent libraries to see if the latest code will work.
It is expected that these will often fail, since the upgrades were previously rejected for failing pipelines and have not been directly addressed yet.
This upgrade should only be merged in the CI pipeline reports success.
If this MR has failed, we can spend a little time investigating to see if a trivial upgrade could achieve compatiblity to the new library.
But significant upgrade efforts should not occur on this MR, as part of the release tagging process.
Instead, significant work should be scheduled for a subsequent milestone.
This MR may co-exist with a separate, smaller upgrade MR.
If both pass, this one should be used instead.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: 47a0054bec9bb13be125f218c8f010a55af61dec
Maven: 0.26.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | ---------- | ------------------ |
| core-lib-azure | 0.25.0-rc2 | |
| core-lib-gc | 0.24.0 | |
| os-core-lib-aws | 0.25.0-rc3 | 0.25.0-rc3 |
| obm | 0.24.0 | |
| oqm | 0.24.0 | |
| os-core-common | 0.25.0-rc3 | 0.25.0-rc3, 0.24.0 |
| os-core-lib-ibm | 0.24.0 | 0.24.0 |
| osm | 0.24.0 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.0 | 2.8.1, 2.13.2.2 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.30, 2.0 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade
SHA: 754c7abbbd0c8cd7450616bf784b51cbf15d7c22
Maven: 0.26.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | ------ | --------------- |
| core-lib-azure | 0.25.0 | |
| core-lib-gc | 0.25.0 | |
| os-core-lib-aws | 0.25.0 | 0.25.0 |
| obm | 0.25.0 | |
| oqm | 0.25.0 | |
| os-core-common | 0.25.0 | 0.25.0 |
| os-core-lib-ibm | 0.25.0 | 0.25.0 |
| osm | 0.25.0 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.0 | 2.8.1, 2.13.2.2 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.30, 2.0 |M22 - Release 0.25https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/807Cherry-pick 'Update persistablereference by UnitOfMeasureID' into release/0.252023-12-14T10:22:11ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Update persistablereference by UnitOfMeasureID' into release/0.25**Original MR**: !796
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !796
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/storage/-/pipelines/new?ref=cherry-pick-for-796)M22 - Release 0.25David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/804Cherry-pick '[MSOSDU-1787]: Increase code coverage percentage' into release/0.252023-12-12T10:20:50ZDavid Diederichd.diederich@opengroup.orgCherry-pick '[MSOSDU-1787]: Increase code coverage percentage' into release/0.25**Original MR**: !800
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !800
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/storage/-/pipelines/new?ref=cherry-pick-for-800)M22 - Release 0.25David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/800[MSOSDU-1787]: Increase code coverage percentage2023-12-12T10:20:11ZDeepa Kumari[MSOSDU-1787]: Increase code coverage percentageCode coverage increment for storage azure:
* Issue Reference: https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/40
* Added `apache-maven-plugin` dependency to fix the jacoco report generation
* Added exclusions for ...Code coverage increment for storage azure:
* Issue Reference: https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/issues/40
* Added `apache-maven-plugin` dependency to fix the jacoco report generation
* Added exclusions for the classes that were not required to be evaluated for tests coverage report in maven plugin exclusion list
* Generated Code Coverage [Report](https://osdu.pages.opengroup.org/-/platform/system/storage/-/jobs/2447260/artifacts/provider/storage-azure/target/site/jacoco/index.html)
**Initial code coverage: 16%**
**Current code coverage: 67%**M22 - Release 0.25Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/805Cherry-pick '[MSCOSDU-1867] fix: upgrade netty and org.json' into release/0.252023-12-12T10:19:46ZDavid Diederichd.diederich@opengroup.orgCherry-pick '[MSCOSDU-1867] fix: upgrade netty and org.json' into release/0.25**Original MR**: !801
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !801
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/storage/-/pipelines/new?ref=cherry-pick-for-801)M22 - Release 0.25David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/803cherry-pick MR 802 from master2023-12-12T08:45:20ZYunhua Koglincherry-pick MR 802 from masterOriginal MR: [!802](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/802) (merged)Original MR: [!802](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/802) (merged)M22 - Release 0.25Yunhua KoglinYunhua Koglinhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/801[MSCOSDU-1867] fix: upgrade netty and org.json2023-12-12T04:05:02ZDeepa Kumari[MSCOSDU-1867] fix: upgrade netty and org.json1. core-lib-azure: from 0.24.0 to 0.25.0-rc2, includes org.json: from 20220924 to 20231013
1. netty: from 4.1.98 to 4.1.1011. core-lib-azure: from 0.24.0 to 0.25.0-rc2, includes org.json: from 20220924 to 20231013
1. netty: from 4.1.98 to 4.1.101M22 - Release 0.25Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/802disable mongodb2023-12-11T19:54:12ZYunhua Koglindisable mongodb# Merge request template# Merge request templateM22 - Release 0.25Yunhua KoglinYunhua Koglinhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/799Remove SNAPSHOT dependencies2023-12-08T23:57:08ZDavid Diederichd.diederich@opengroup.orgRemove SNAPSHOT dependenciesThis automated MR removes usage of `SNAPSHOT` versions in the first party library dependencies.
Since `SNAPSHOT` dependencies change frequently -- by their nature -- usage of them across projects is dangerous and should be avoided.
### ...This automated MR removes usage of `SNAPSHOT` versions in the first party library dependencies.
Since `SNAPSHOT` dependencies change frequently -- by their nature -- usage of them across projects is dangerous and should be avoided.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: a99aefe42c2091e0e0032c6ed924c43472c483d7
Maven: 0.25.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | --------------- | ------------------ |
| core-lib-azure | 0.24.0 | |
| core-lib-gc | 0.24.0 | |
| os-core-lib-aws | 0.25.0-SNAPSHOT | 0.25.0-SNAPSHOT |
| obm | 0.24.0 | |
| oqm | 0.24.0 | |
| os-core-common | 0.25.0-rc3 | 0.25.0-rc3, 0.24.0 |
| os-core-lib-ibm | 0.24.0 | 0.24.0 |
| osm | 0.24.0 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.0 | 2.8.1, 2.13.2.2 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.30, 2.0 |
### Dependency Information After the Upgrade
```
Branch: snapshot-removal
SHA: 6221c7ec78db144477ff8e3e2be489095f231b78
Maven: 0.25.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | ---------- | ------------------ |
| core-lib-azure | 0.24.0 | |
| core-lib-gc | 0.24.0 | |
| os-core-lib-aws | 0.25.0-rc3 | 0.25.0-rc3 |
| obm | 0.24.0 | |
| oqm | 0.24.0 | |
| os-core-common | 0.25.0-rc3 | 0.25.0-rc3, 0.24.0 |
| os-core-lib-ibm | 0.24.0 | 0.24.0 |
| osm | 0.24.0 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.0 | 2.8.1, 2.13.2.2 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.30, 2.0 |M22 - Release 0.25https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/790Add user and time properties to Record model2023-12-07T03:00:27ZNeha KhandelwalAdd user and time properties to Record modelUpdated os-core-common version to 0.25.0-rc3 to include the new properties in the Record model.
Related issue https://community.opengroup.org/osdu/platform/system/storage/-/issues/184Updated os-core-common version to 0.25.0-rc3 to include the new properties in the Record model.
Related issue https://community.opengroup.org/osdu/platform/system/storage/-/issues/184M22 - Release 0.25https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/795[MSCOSDU-1095] Config : Azure container name configuration2023-12-06T05:06:51ZThulasi Dass Subramanian[MSCOSDU-1095] Config : Azure container name configuration# Change Details
- Azure container name configuration with default value from environment# Change Details
- Azure container name configuration with default value from environmentM22 - Release 0.25Thulasi Dass SubramanianThulasi Dass Subramanianhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/797fix aws versions2023-12-05T20:49:17ZYunhua Koglinfix aws versions# Merge request template# Merge request templateM22 - Release 0.25Yunhua KoglinYunhua Koglinhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/792Update dependancy version to fix vulnerabilities2023-12-05T10:01:11ZSolomon AyalewUpdate dependancy version to fix vulnerabilities- pull latest version os-core-common
- fix CVE-2023-20863,CVE-2023-20883, CVE-2023-34034 and other org.springframework:spring-core/security vulnerabilities- pull latest version os-core-common
- fix CVE-2023-20863,CVE-2023-20883, CVE-2023-34034 and other org.springframework:spring-core/security vulnerabilitiesM22 - Release 0.25https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/785Fixed Update/Patch apis Data Authorization Issues2023-12-01T01:55:19ZDadong ZhouFixed Update/Patch apis Data Authorization Issues## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
https://community.opengroup.org/osdu/platform/system/storage/-/issues/174
https://community.opengroup.org/osdu...## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
https://community.opengroup.org/osdu/platform/system/storage/-/issues/174
https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/500
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
## What is the new/expected behavior?
## Have you added/updated Unit Tests and Integration Tests?
## Any other useful informationM22 - Release 0.25Dadong ZhouDadong Zhouhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/791AWS - Merge from dev2023-11-27T19:39:31ZGuillaume CailletAWS - Merge from devOnly AWS specific changes.
* Add or update License header
* Upgrade AWS core lib
* Upgrade AWS dependencies to fix CVEsOnly AWS specific changes.
* Add or update License header
* Upgrade AWS core lib
* Upgrade AWS dependencies to fix CVEsM22 - Release 0.25Guillaume CailletGuillaume Caillethttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/786fix duplicate entries while logging2023-11-13T09:21:53ZVidyaDharani Lokamfix duplicate entries while loggingRelated Issue: https://community.opengroup.org/osdu/platform/system/storage/-/issues/182
As mentioned in the above issue we have same operation_id for disparate operations(based on record ids).
- In this MR enabled sampling for applicat...Related Issue: https://community.opengroup.org/osdu/platform/system/storage/-/issues/182
As mentioned in the above issue we have same operation_id for disparate operations(based on record ids).
- In this MR enabled sampling for application insights to reduce the duplicacy in operation id.
- Logs of QueryApi/getRecords API
![image](/uploads/d71f759d8718d0789bfdf437e9ba2d29/image.png)
- Logs of QueryApi/fetchRecords API
![image](/uploads/a2539327cc09043b8e3e3bfde373542a/image.png)M22 - Release 0.25VidyaDharani LokamVidyaDharani Lokamhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/783[Azure] Fix `Get Record by version` 500 Error for non matching version2023-10-25T14:53:24ZThulasi Dass Subramanian[Azure] Fix `Get Record by version` 500 Error for non matching version# Change Description:
* Issue reference: https://community.opengroup.org/osdu/platform/system/storage/-/issues/181
* **Get Record by version:** Fix GcsVersionPath checking condition from `contains` to `endsWith` for given `version`, to ...# Change Description:
* Issue reference: https://community.opengroup.org/osdu/platform/system/storage/-/issues/181
* **Get Record by version:** Fix GcsVersionPath checking condition from `contains` to `endsWith` for given `version`, to find the exact match of version path.
* Below Response - `HTTP 404 status` with reason will be returned _(instead of `HTTP 50O status`)_
```plaintext
{
"code": 404,
"reason": "Version not found",
"message": "The version 2 can't be found for record opendes:master-data--Well:999310036787"
}
```
# Changes in:
* [x] Azure
# Dev Checklist:
* [x] Added Unit Tests, wherever applicable.
* [x] Existing Tests pass
* [x] Verified functionality locally
* [x] Self Reviewed my code for formatting and complex business logic.
* [x] renamed few methods & variables for better readabilityM22 - Release 0.25Thulasi Dass SubramanianThulasi Dass Subramanian