Storage merge requestshttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests2024-03-19T17:13:58Zhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/833enable collaboration header2024-03-19T17:13:58ZYunhua Koglinenable collaboration header# Merge request template# Merge request templateM23 - Release 0.26Yunhua KoglinYunhua Koglinhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/829Fix vulnerbilities2024-01-23T14:38:11ZBruce JinFix vulnerbilitiesFix existing critical CVEs and other Vulnerabilities:
https://community.opengroup.org/osdu/platform/system/storage/-/issues/196
https://community.opengroup.org/osdu/platform/system/storage/-/issues/194
https://community.opengroup.org/osd...Fix existing critical CVEs and other Vulnerabilities:
https://community.opengroup.org/osdu/platform/system/storage/-/issues/196
https://community.opengroup.org/osdu/platform/system/storage/-/issues/194
https://community.opengroup.org/osdu/platform/system/storage/-/issues/197
https://community.opengroup.org/osdu/platform/system/storage/-/issues/198
https://community.opengroup.org/osdu/platform/system/storage/-/issues/199
https://community.opengroup.org/osdu/platform/system/storage/-/issues/200
https://community.opengroup.org/osdu/platform/system/storage/-/issues/201
https://community.opengroup.org/osdu/platform/system/storage/-/issues/202
https://community.opengroup.org/osdu/platform/system/storage/-/issues/203M23 - Release 0.26Bruce JinBruce Jinhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/828[MSCOSDU-1979] fix code coverage2024-01-18T09:10:28ZDeepa Kumari[MSCOSDU-1979] fix code coverageIncrease the code coverage in storage service from 67% to 91%Increase the code coverage in storage service from 67% to 91%M23 - Release 0.26Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/827remove aws integration test override2024-01-22T17:20:54ZYunhua Koglinremove aws integration test override# Merge request template# Merge request templateM23 - Release 0.26Yunhua KoglinYunhua Koglinhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/826Refactoring code for Reference Implemetat2024-02-23T12:25:47ZAshutosh KumarRefactoring code for Reference ImplemetatLink to story: https://gitlab.opengroup.org/osdu/pmc/community-implementation/-/issues/10
More details about the solution: https://community.opengroup.org/osdu/documentation/-/wikis/Pluggable-approach-for-Mappers-and-Drivers
As a part ...Link to story: https://gitlab.opengroup.org/osdu/pmc/community-implementation/-/issues/10
More details about the solution: https://community.opengroup.org/osdu/documentation/-/wikis/Pluggable-approach-for-Mappers-and-Drivers
As a part of the community implementation solution GC module was refactored to the Core-Plus module, in addition new pluggable approach for Drivers was used.
## What is the current behavior?
* We do not have the core-plus module.
* Drivers not pluggable, all dependencies bounded during the compile step.
## What is the new behavior?
* We have the core-plus module.
* Drivers are pluggable and could be attached to service during image build, not compile.
## Dev Checklist:
* Start
* [x] Copy **./provider/service-gc** to **[./service-core-plus](https://community.opengroup.org/osdu/platform/system/dataset/-/tree/master/dataset-core-plus?ref_type=heads)**
* Dependencies
* [x] Add `core-plus` as a module to the root pom file, located in **[./pom.xml](https://community.opengroup.org/osdu/platform/system/dataset/-/blob/master/pom.xml?ref_type=heads#L232)**
* [x] Replace Google Drivers OQM, OBM, OSM with latest Community Core Drivers OQM, OBM, OSM, ex: old **[oqm](https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/oqm)** new **[os-oqm-core](https://community.opengroup.org/osdu/platform/system/lib/drivers/os-oqm)**
* [x] Replace **[core-lib-gc](https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/os-core-lib-gcp)** with **[core-plus-common-lib](https://community.opengroup.org/osdu/platform/system/lib/core-plus/core-plus-common-lib)**
* [x] Add **[spring boot loader](https://community.opengroup.org/osdu/platform/system/dataset/-/blob/master/dataset-core-plus/pom.xml#L54)** dependency.
* [x] Update `spring-boot-maven` plugin, to use **[PropertiesLauncher](https://community.opengroup.org/osdu/platform/system/dataset/-/blob/master/dataset-core-plus/pom.xml#L170)** as a main class.
* [x] Remove Google Cloud dependencies, except Google Datastore (it's a known **[issue](https://gitlab.opengroup.org/osdu/pmc/community-implementation/-/issues/26)**. Datastore dependency should accompany OSM, if you had to add it please update the issue with the project link).M23 - Release 0.26Ashutosh KumarAshutosh Kumarhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/825Reject record ids with unsupported characters at the end2024-01-31T23:00:32ZNeha KhandelwalReject record ids with unsupported characters at the endRelated issue #213.
For Storage create/update record API, if a record ID ends with a dot (.), a forward slash (/), or a backslash (\\) the data block for the record is not properly uploaded to the Microsoft storage account. Create/updat...Related issue #213.
For Storage create/update record API, if a record ID ends with a dot (.), a forward slash (/), or a backslash (\\) the data block for the record is not properly uploaded to the Microsoft storage account. Create/update record API should reject record IDs that end in these unsupported characters (i.e. return 400 bad request when such record IDs are used).M23 - Release 0.26https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/824AWS collaboration header support2024-01-23T14:35:50ZMark ChanceAWS collaboration header support# Merge request template# Merge request templateM23 - Release 0.26https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/819Add support more than 10 patch operations in a single request2024-01-19T16:12:19ZYurii KondakovAdd support more than 10 patch operations in a single request## Type of change
- [x] Bug Fix
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [X] Azure
- [ ] Google Cloud
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
Currently,...## Type of change
- [x] Bug Fix
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [X] Azure
- [ ] Google Cloud
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
Currently, we have a limit of 10 operations on the maximum number of patch operations due to limitations on the part of Azure Cosmos DB.
[Partial document update in Azure Cosmos DB](https://learn.microsoft.com/en-us/azure/cosmos-db/partial-document-update)
For the API consumers, this number is less by 2, i.e. 8, as we add two more patch operations internally (at storage service).
The essence of the changes is that if the number of operations is more than 10,then we divide them into portions of a maximum of 10 operations.
This merge request is a second part of Bug fix. The first part is changes in the core-lib-azure library
https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/321M23 - Release 0.26Yurii KondakovYurii Kondakovhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/818Gonrg 9065 storage bootstrap disable2024-01-17T09:44:08ZAliaksandr Ramanovich (EPAM)Gonrg 9065 storage bootstrap disable
GC changes only
Disable storage bootstrap due to duplicating this process in Data bootstrap repo
GC changes only
Disable storage bootstrap due to duplicating this process in Data bootstrap repoM23 - Release 0.26Aliaksandr Ramanovich (EPAM)Aliaksandr Ramanovich (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/814add liveness_check endpoint for azure2024-01-04T07:34:36Zsaketh somarajuadd liveness_check endpoint for azure- MR Reference: https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/793
- Added `liveness_check` endpoint in collaboration filter for azure.- MR Reference: https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/793
- Added `liveness_check` endpoint in collaboration filter for azure.M23 - Release 0.26VidyaDharani LokamVidyaDharani Lokamhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/813[MSCOSDU-1894] upgrade reactor-netty, log4j2, okhttp and okio2024-01-12T00:21:10ZVidyaDharani Lokam[MSCOSDU-1894] upgrade reactor-netty, log4j2, okhttp and okio# Change details
### Common code
* upgraded `log4j2` version to `2.22.0`
### Azure
* upgraded `reactor-netty` related dependencies to version `1.1.14`
* upgraded `nimbus-jose-jwt-azure` to `9.30.2`.
* remediate `container_scanning` f...# Change details
### Common code
* upgraded `log4j2` version to `2.22.0`
### Azure
* upgraded `reactor-netty` related dependencies to version `1.1.14`
* upgraded `nimbus-jose-jwt-azure` to `9.30.2`.
* remediate `container_scanning` flagged vulnerabilities `log4j2,okio,okhttp`
# Changes in:
* [x] GCP
* [x] Azure
* [x] AWS
* [x] IBMM23 - Release 0.26VidyaDharani LokamVidyaDharani Lokamhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/812Change Cosmos DB service unavailable error code to 5032024-01-03T20:55:00ZNeha KhandelwalChange Cosmos DB service unavailable error code to 503Currently, Storage batch API responds with 500 error code due to ServiceUnavailableException on CosmosDB. Indexer service, which uses Storage batch API, does not retry if downstream services are responding with 500 error code. Storage ba...Currently, Storage batch API responds with 500 error code due to ServiceUnavailableException on CosmosDB. Indexer service, which uses Storage batch API, does not retry if downstream services are responding with 500 error code. Storage batch API should respond with 503 error code when CosmosDB service cannot be reached to allow Indexer to retry.M23 - Release 0.26https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/810Use full URL instead of relative path2024-02-09T10:46:46ZLawrence ChanUse full URL instead of relative path* Issue Reference: [#68](https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/issues/68)
* Added configuration `api.server.fullUrl.enabled` to enable full server url in OpenAPI swagger
* Currently only in Azure i...* Issue Reference: [#68](https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/issues/68)
* Added configuration `api.server.fullUrl.enabled` to enable full server url in OpenAPI swagger
* Currently only in Azure it is enabled. For Other \[CSP/Common Core\] there is no change.
### Configuration Details
* `api.server.fullUrl.enabled=true` It will generate full server url in the OpenAPI swagger
* `api.server.fullUrl.enabled=false` It will generate only the contextPath
* Reference: https://springdoc.org/faq.html#_how_is_server_url_generatedM23 - Release 0.26https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/809Gonrg 8958 disable policy2023-12-28T13:29:02ZSiarhei Poliak [EPAM / GCP]Gonrg 8958 disable policy# Merge request template# Merge request templateM23 - Release 0.26https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/808Cherry-pick 'Full Upgrade of First Party Library Dependencies for Release 0.2...2023-12-16T10:24:13ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Full Upgrade of First Party Library Dependencies for Release 0.25' into release/0.25**Original MR**: !806
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !806
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/storage/-/pipelines/new?ref=cherry-pick-for-806)M22 - Release 0.25David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/807Cherry-pick 'Update persistablereference by UnitOfMeasureID' into release/0.252023-12-14T10:22:11ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Update persistablereference by UnitOfMeasureID' into release/0.25**Original MR**: !796
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !796
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/storage/-/pipelines/new?ref=cherry-pick-for-796)M22 - Release 0.25David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/806Full Upgrade of First Party Library Dependencies for Release 0.252023-12-15T20:06:24ZDavid Diederichd.diederich@opengroup.orgFull Upgrade of First Party Library Dependencies for Release 0.25This generated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to try to fully upgrade all dependent libraries to see if the latest code will work.
It is expected that these will ...This generated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to try to fully upgrade all dependent libraries to see if the latest code will work.
It is expected that these will often fail, since the upgrades were previously rejected for failing pipelines and have not been directly addressed yet.
This upgrade should only be merged in the CI pipeline reports success.
If this MR has failed, we can spend a little time investigating to see if a trivial upgrade could achieve compatiblity to the new library.
But significant upgrade efforts should not occur on this MR, as part of the release tagging process.
Instead, significant work should be scheduled for a subsequent milestone.
This MR may co-exist with a separate, smaller upgrade MR.
If both pass, this one should be used instead.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: 47a0054bec9bb13be125f218c8f010a55af61dec
Maven: 0.26.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | ---------- | ------------------ |
| core-lib-azure | 0.25.0-rc2 | |
| core-lib-gc | 0.24.0 | |
| os-core-lib-aws | 0.25.0-rc3 | 0.25.0-rc3 |
| obm | 0.24.0 | |
| oqm | 0.24.0 | |
| os-core-common | 0.25.0-rc3 | 0.25.0-rc3, 0.24.0 |
| os-core-lib-ibm | 0.24.0 | 0.24.0 |
| osm | 0.24.0 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.0 | 2.8.1, 2.13.2.2 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.30, 2.0 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade
SHA: 754c7abbbd0c8cd7450616bf784b51cbf15d7c22
Maven: 0.26.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | ------ | --------------- |
| core-lib-azure | 0.25.0 | |
| core-lib-gc | 0.25.0 | |
| os-core-lib-aws | 0.25.0 | 0.25.0 |
| obm | 0.25.0 | |
| oqm | 0.25.0 | |
| os-core-common | 0.25.0 | 0.25.0 |
| os-core-lib-ibm | 0.25.0 | 0.25.0 |
| osm | 0.25.0 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.0 | 2.8.1, 2.13.2.2 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.30, 2.0 |M22 - Release 0.25https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/805Cherry-pick '[MSCOSDU-1867] fix: upgrade netty and org.json' into release/0.252023-12-12T10:19:46ZDavid Diederichd.diederich@opengroup.orgCherry-pick '[MSCOSDU-1867] fix: upgrade netty and org.json' into release/0.25**Original MR**: !801
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !801
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/storage/-/pipelines/new?ref=cherry-pick-for-801)M22 - Release 0.25David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/804Cherry-pick '[MSOSDU-1787]: Increase code coverage percentage' into release/0.252023-12-12T10:20:50ZDavid Diederichd.diederich@opengroup.orgCherry-pick '[MSOSDU-1787]: Increase code coverage percentage' into release/0.25**Original MR**: !800
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !800
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/storage/-/pipelines/new?ref=cherry-pick-for-800)M22 - Release 0.25David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/803cherry-pick MR 802 from master2023-12-12T08:45:20ZYunhua Koglincherry-pick MR 802 from masterOriginal MR: [!802](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/802) (merged)Original MR: [!802](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/802) (merged)M22 - Release 0.25Yunhua KoglinYunhua Koglin