Storage merge requestshttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests2023-08-18T22:23:16Zhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/241Resolve: Gonrg 2833 storage on gke2023-08-18T22:23:16ZMykola Zamkovyi (EPAM)Resolve: Gonrg 2833 storage on gke# Merge request template# Merge request templateM8 - Release 0.11https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/243response content-type as application/json not text/plain. use media-types2023-08-18T22:23:14ZNeelesh Thakurresponse content-type as application/json not text/plain. use media-typesaddresses #85addresses #85M8 - Release 0.11https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/245Update resources limits [GONRG-2974]2023-08-18T22:23:12ZMikhail Piatliou (EPAM)Update resources limits [GONRG-2974]# Merge request template# Merge request templateM8 - Release 0.11Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/249update core-lib-azure2023-08-18T22:23:11ZAlok Joshiupdate core-lib-azureIncrement core-lib-azure library to incorporate some caching [improvements](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/131) on KV accessIncrement core-lib-azure library to incorporate some caching [improvements](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/131) on KV accessM8 - Release 0.11Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/253handle erroneous AsIngestedCoordinates payload2023-08-18T22:23:09ZNeelesh Thakurhandle erroneous AsIngestedCoordinates payloadHandle erroneous AsIngestedCoordinates payload, Storage batch request fails with 500 error code for following payload:
```json
"data": {
"SpatialLocation": {
"AsIngestedCoordinates": {
"Type": "AnyCrsFeatureCollection"...Handle erroneous AsIngestedCoordinates payload, Storage batch request fails with 500 error code for following payload:
```json
"data": {
"SpatialLocation": {
"AsIngestedCoordinates": {
"Type": "AnyCrsFeatureCollection",
"CoordinateReferenceSystemID": "NAD27",
"VerticalCoordinateReferenceSystemID": null,
"PersistableReferenceCrs": null,
"PersistableReferenceVerticalCrs": null,
"PersistableReferenceUnitZ": null,
"Features": [
{
"Type": "AnyCrsFeature",
"Properties": null,
"Geometry": {
"Coordinates": [
30.8793048,
-87.7801454
],
"Type": "AnyCrsPoint",
"Bbox": null
}
}
],
"Bbox": null
},
"Wgs84Coordinates": null
}
}
```M8 - Release 0.11https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/254Provide default value false to schema endpoint config2023-08-18T22:23:08ZKrishna Nikhil VedurumudiProvide default value false to schema endpoint config## Type of change
- [ ] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [YES/NO]
## Does this introduce a change in the cloud p...## Type of change
- [ ] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [YES/NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [YES/NO]
## What is the current behavior?
## What is the new/expected behavior?
Provide default value to schema endpoint disablement feature flag.
## Have you added/updated Unit Tests and Integration Tests?
## Any other useful informationM8 - Release 0.11https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/256AWS Fixes2023-08-18T22:23:06ZMatt WiseAWS Fixes# Merge request template# Merge request templateM8 - Release 0.11Matt WiseMatt Wisehttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/257integration tests fix for schema api feature flag2023-08-18T22:23:04ZLarissa Pereiraintegration tests fix for schema api feature flagRelated ADR: [62](https://community.opengroup.org/osdu/platform/system/storage/-/issues/62)
**Background** Instead of complete irreversible removal of the Storage Schema endpoints, a fail-safe option to restore application functionality...Related ADR: [62](https://community.opengroup.org/osdu/platform/system/storage/-/issues/62)
**Background** Instead of complete irreversible removal of the Storage Schema endpoints, a fail-safe option to restore application functionality is needed in cases where missed dependencies on Storage Schemas APIs have gone undetected.
This MR fixes any integration test failures due to the feature flag by skipping the tests if the feature flag has been disabled (viz. schema endpoints are disabled)M10 - Release 0.13https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/261Default env variable value for schema endpoint2023-08-18T22:23:02ZKrishna Nikhil VedurumudiDefault env variable value for schema endpoint## Type of change
- [ ] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provi...## Type of change
- [ ] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [YES/NO]
## What is the current behavior?
Due to lack of default value, pipelines are broken with Storage service not coming up with error
```
***************************
APPLICATION FAILED TO START
***************************
Description:
Failed to bind properties under 'schema.endpoints.disabled' to boolean:
Property: schema.endpoints.disabled
Value:
Origin: System Environment Property "schema_endpoints_disabled"
Reason: failed to convert java.lang.String to boolean
Action:
Update your application's configuration
```
## What is the new/expected behavior?
Default value provided for application property `schema.endpoints.disabled`
## Have you added/updated Unit Tests and Integration Tests?
## Any other useful informationM9 - Release 0.12Krishna Nikhil VedurumudiKrishna Nikhil Vedurumudihttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/262Mr fix trusted2023-08-18T22:23:01ZRitika KaushalMr fix trusted# Merge request template
reverted few rebase mistakes# Merge request template
reverted few rebase mistakesM9 - Release 0.12Ritika KaushalRitika Kaushalhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/264fix issue with impossibility of retrieving version with changed kind2023-08-18T22:22:59ZYauheni Lesnikaufix issue with impossibility of retrieving version with changed kindFix the issue when the version can't be retrieved because of changed kind.
Issue: https://community.opengroup.org/osdu/platform/system/storage/-/issues/89Fix the issue when the version can't be retrieved because of changed kind.
Issue: https://community.opengroup.org/osdu/platform/system/storage/-/issues/89M9 - Release 0.12https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/265Handle invalid coordinates for AsIngestedCoordinates payload2023-08-18T22:22:58ZSanjeev-SLBHandle invalid coordinates for AsIngestedCoordinates payloadHandle invalid coordinates dimension size for AsIngestedCoordinates payload, fixed Storage batch request 5xx errors for the following payload:
`"data": {``
"Name": "MultiPolygon",
"SpatialLocation": {
"AsIngestedCoordinates": {
"...Handle invalid coordinates dimension size for AsIngestedCoordinates payload, fixed Storage batch request 5xx errors for the following payload:
`"data": {``
"Name": "MultiPolygon",
"SpatialLocation": {
"AsIngestedCoordinates": {
"features": [{
"geometry": {
"coordinates": [
[
[
[
313405.9477893702,
6544797.620047403
],
[
314405.9477893702,
6564797.620047403
],
[
312405.9477893702,
6554797.620047403
],
[
313405.9477893702,
6544797.620047403
]
]
],
[
[
313405.9477893702,
6544797.620047403
],
[
324405.9477893702,
6664797.620047403
],
[
322405.9477893702,
6654797.620047403
],
[
313405.9477893702,
6544797.620047403
]
]
],
"bbox": null,
"type": "AnyCrsMultiPolygon"
},
"bbox": null,
"properties": {},
"type": "AnyCrsFeature"
}],
"bbox": null,
"properties": {},
"persistableReferenceCrs": "{\"lateBoundCRS\":{\"wkt\":\"PROJCS[\\\"ED_1950_UTM_Zone_32N\\\",GEOGCS[\\\"GCS_European_1950\\\",DATUM[\\\"D_European_1950\\\",SPHEROID[\\\"International_1924\\\",6378388.0,297.0]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],PROJECTION[\\\"Transverse_Mercator\\\"],PARAMETER[\\\"False_Easting\\\",500000.0],PARAMETER[\\\"False_Northing\\\",0.0],PARAMETER[\\\"Central_Meridian\\\",9.0],PARAMETER[\\\"Scale_Factor\\\",0.9996],PARAMETER[\\\"Latitude_Of_Origin\\\",0.0],UNIT[\\\"Meter\\\",1.0],AUTHORITY[\\\"EPSG\\\",23032]]\",\"ver\":\"PE_10_3_1\",\"name\":\"ED_1950_UTM_Zone_32N\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"23032\"},\"type\":\"LBC\"},\"singleCT\":{\"wkt\":\"GEOGTRAN[\\\"ED_1950_To_WGS_1984_23\\\",GEOGCS[\\\"GCS_European_1950\\\",DATUM[\\\"D_European_1950\\\",SPHEROID[\\\"International_1924\\\",6378388.0,297.0]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],METHOD[\\\"Position_Vector\\\"],PARAMETER[\\\"X_Axis_Translation\\\",-116.641],PARAMETER[\\\"Y_Axis_Translation\\\",-56.931],PARAMETER[\\\"Z_Axis_Translation\\\",-110.559],PARAMETER[\\\"X_Axis_Rotation\\\",0.893],PARAMETER[\\\"Y_Axis_Rotation\\\",0.921],PARAMETER[\\\"Z_Axis_Rotation\\\",-0.917],PARAMETER[\\\"Scale_Difference\\\",-3.52],AUTHORITY[\\\"EPSG\\\",1612]]\",\"ver\":\"PE_10_3_1\",\"name\":\"ED_1950_To_WGS_1984_23\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"1612\"},\"type\":\"ST\"},\"ver\":\"PE_10_3_1\",\"name\":\"ED50 * EPSG-Nor N62 2001 / UTM zone 32N [23032,1612]\",\"authCode\":{\"auth\":\"SLB\",\"code\":\"23032023\"},\"type\":\"EBC\"}",
"persistableReferenceUnitZ": "{\"scaleOffset\":{\"scale\":1.0,\"offset\":0.0},\"symbol\":\"m\",\"baseMeasurement\":{\"ancestry\":\"Length\",\"type\":\"UM\"},\"type\":\"USO\"}",
"type": "AnyCrsFeatureCollection"
}
}
}
}`M9 - Release 0.12Sanjeev-SLBSanjeev-SLBhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/266Usage of publish facade2023-08-18T22:22:56ZNikhil Singh[MicroSoft]Usage of publish facade## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NO] Does the MR contain pipeline/ helm chart related changes?
* [NA] I have u...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NO] Does the MR contain pipeline/ helm chart related changes?
* [NA] I have updated the documentation accordingly.
* [YES] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [NA] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
This MR introduces the use of publisher facade which can be used to publish messages to message brokers(Service Bus/ Event Grid).The pub sub configuration can be used to configure publishing for Event grid and Service bus. The batch size is taken as "10" for both Event Grid and Service bus. Earlier they were 10 and 50 respectively.
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
- All existing and new TC's passed.
## Does this introduce a breaking change?
-------------------------------------
- [NO]M11 - Release 0.14Nikhil Singh[MicroSoft]Nikhil Singh[MicroSoft]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/267Add Virtual Service2023-08-18T22:22:54ZSanjeev-SLBAdd Virtual ServiceIssue [32](https://community.opengroup.org/osdu/platform/system/notification/-/issues/32)Issue [32](https://community.opengroup.org/osdu/platform/system/notification/-/issues/32)M9 - Release 0.12Sanjeev-SLBSanjeev-SLBhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/268bulk soft delete record api2023-08-18T22:22:53ZYauheni Lesnikaubulk soft delete record apiNew API to perform soft delete for bulk of records. Related ADR: https://community.opengroup.org/osdu/platform/system/storage/-/issues/83New API to perform soft delete for bulk of records. Related ADR: https://community.opengroup.org/osdu/platform/system/storage/-/issues/83M9 - Release 0.12Yauheni LesnikauYauheni Lesnikauhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/269hash cosmodb cursor as it's too large causing issue for Istio envoy & Azure A...2023-08-18T22:22:50ZNeelesh Thakurhash cosmodb cursor as it's too large causing issue for Istio envoy & Azure App GateWayAddresses #[34](https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/34)Addresses #[34](https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/34)M9 - Release 0.12https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/271respond with 503 for too many cosmosdb request2023-08-18T22:22:48ZNeelesh Thakurrespond with 503 for too many cosmosdb requestM9 - Release 0.12https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/272broken pipe IOException handler2023-08-18T22:22:47ZNeelesh Thakurbroken pipe IOException handlerStorage Service is throwing IOException with `Broken pipe` error message. This usually thrown when a client closes the connection while its request is still being processed in the server.
We should handle these properly so client can re...Storage Service is throwing IOException with `Broken pipe` error message. This usually thrown when a client closes the connection while its request is still being processed in the server.
We should handle these properly so client can retry.
Here is one of the stack trace:
![image](/uploads/3177e0f0678500da915bcb309725d955/image.png)M9 - Release 0.12https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/273Upgrading core-lib-azure version2023-08-18T22:22:45ZAbhishek PatilUpgrading core-lib-azure version# Merge request template# Merge request templateM9 - Release 0.12Abhishek PatilAbhishek Patilhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/275GONRG-3398 - set default number of replicas for storage2023-08-18T22:22:43ZOleksandr Kosse (EPAM)GONRG-3398 - set default number of replicas for storageScale storage serviceScale storage serviceM9 - Release 0.12Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)