Partition merge requestshttps://community.opengroup.org/osdu/platform/system/partition/-/merge_requests2020-11-20T00:31:43Zhttps://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/22WIP: Update .gitlab-ci.yml to use patch-3 of pipeline.2020-11-20T00:31:43ZHema Vishnu Pola [Microsoft]WIP: Update .gitlab-ci.yml to use patch-3 of pipeline.Verifying k8 rollout status command.Verifying k8 rollout status command.https://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/30WIP: Fixing ADO pipeline to only build azure provider code2021-04-26T15:55:20ZJasonWIP: Fixing ADO pipeline to only build azure provider codeCurrently azure pipeline is passing in the option "-P \[service_name\]-azure" when building the provider code and "-P \[service_name\]-core" when building the core code. The -P flag is meant to indicate what profile to activate. Instead,...Currently azure pipeline is passing in the option "-P \[service_name\]-azure" when building the provider code and "-P \[service_name\]-core" when building the core code. The -P flag is meant to indicate what profile to activate. Instead, we want to be using the -pl argument, which indicates which modules should be built. This will ensure that we are only building Azure code and only caching Azure dependencies.JasonJasonhttps://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/121Vuln fix netty codec2022-01-10T07:25:44ZGokul NagareVuln fix netty codecAnuj GuptaShrikant GargAnuj Guptahttps://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/304upgrading corelib azure for partition service2023-05-13T05:03:42ZNishant Vidyasagarupgrading corelib azure for partition serviceNishant VidyasagarNishant Vidyasagarhttps://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/443Upgrade First Party Library Dependencies for Release 0.232023-09-05T21:05:02ZDavid Diederichd.diederich@opengroup.orgUpgrade First Party Library Dependencies for Release 0.23This generated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any...This generated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any library that is older than the previous release will be left as-is, since the upgrade is likely to be more complicated.
Furthermore, the upgrade should only be merged in the CI pipeline reports success.
If this MR has failed, we can spend a little time investigating to see if a trivial upgrade could achieve compatiblity to the new library.
But significant upgrade efforts should not occur on this MR, as part of the release tagging process.
Instead, significant work should be scheduled for a subsequent milestone.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: ad89dc0b0319810b4a00ff18781aabac452f3142
Maven: 0.24.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| --------------------------------------------------- | --------------- | -------------- |
| core-lib-azure | 0.22.0 | |
| core-lib-gc | 0.23.0 | 0.23.0 |
| os-core-lib-aws | 0.23.0 | 0.23.0 |
| os-core-common | 0.23.1 | 0.23.0 |
| os-core-lib-ibm | 0.23.0 | 0.23.0 |
| osm | 0.23.0 | |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1, 2.17.2 | 2.17.2, 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1, 2.17.2 | 2.17.2, 2.13.3 |
| (3rd Party) org.yaml.snakeyaml | 2.0, 1.33, 1.30 | 1.30, 2.0 |
```
Critical: Found Vulnerable Snake YAML dependency (<2.0)
├─ _Root_
│ ├─ org.opengroup.osdu.partition-ibm == 0.24.0-SNAPSHOT
│ │ └─ org.yaml.snakeyaml == 1.33
│ └─ org.opengroup.osdu.partition-gc == 0.24.0-SNAPSHOT
│ └─ org.springframework.boot.spring-boot-starter-security == 2.7.10
│ └─ org.springframework.boot.spring-boot-starter == 2.7.10
│ └─ org.yaml.snakeyaml == 1.30
└─ testing/
├─ org.opengroup.osdu.partition.partition-test-aws == 0.24.0-SNAPSHOT
│ └─ org.opengroup.osdu.core.aws.os-core-lib-aws == 0.23.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.7.7
│ └─ org.springframework.boot.spring-boot-starter == 2.7.7
│ └─ org.yaml.snakeyaml == 1.30
└─ org.opengroup.osdu.partition.partition-test-gc == 0.24.0-SNAPSHOT
└─ org.opengroup.osdu.core-lib-gc == 0.23.0
└─ org.opengroup.osdu.os-core-common == 0.23.0
└─ org.springframework.boot.spring-boot-starter-web == 2.7.7
└─ org.springframework.boot.spring-boot-starter == 2.7.7
└─ org.yaml.snakeyaml == 1.30
```
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade
SHA: 3edfdd206e1287b8e532b960b505479fc4b11a3d
Maven: 0.24.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| --------------------------------------------------- | --------------- | -------------- |
| core-lib-azure | 0.23.1 | |
| core-lib-gc | 0.23.0 | 0.23.0 |
| os-core-lib-aws | 0.23.0 | 0.23.0 |
| os-core-common | 0.23.2 | 0.23.0 |
| os-core-lib-ibm | 0.23.0 | 0.23.0 |
| osm | 0.23.0 | |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1, 2.17.2 | 2.17.2, 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1, 2.17.2 | 2.17.2, 2.13.3 |
| (3rd Party) org.yaml.snakeyaml | 2.0, 1.33, 1.30 | 1.30, 2.0 |
```
Critical: Found Vulnerable Snake YAML dependency (<2.0)
├─ _Root_
│ ├─ org.opengroup.osdu.partition-ibm == 0.24.0-SNAPSHOT
│ │ └─ org.yaml.snakeyaml == 1.33
│ └─ org.opengroup.osdu.partition-gc == 0.24.0-SNAPSHOT
│ └─ org.springframework.boot.spring-boot-starter-security == 2.7.10
│ └─ org.springframework.boot.spring-boot-starter == 2.7.10
│ └─ org.yaml.snakeyaml == 1.30
└─ testing/
├─ org.opengroup.osdu.partition.partition-test-aws == 0.24.0-SNAPSHOT
│ └─ org.opengroup.osdu.core.aws.os-core-lib-aws == 0.23.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.7.7
│ └─ org.springframework.boot.spring-boot-starter == 2.7.7
│ └─ org.yaml.snakeyaml == 1.30
└─ org.opengroup.osdu.partition.partition-test-gc == 0.24.0-SNAPSHOT
└─ org.opengroup.osdu.core-lib-gc == 0.23.0
└─ org.opengroup.osdu.os-core-common == 0.23.0
└─ org.springframework.boot.spring-boot-starter-web == 2.7.7
└─ org.springframework.boot.spring-boot-starter == 2.7.7
└─ org.yaml.snakeyaml == 1.30
```M20 - Release 0.23https://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/344Upgrade First Party Library Dependencies for Release 0.192023-02-18T07:31:15ZDavid Diederichd.diederich@opengroup.orgUpgrade First Party Library Dependencies for Release 0.19This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any...This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any library that is older than the previous release will be left as-is, since the upgrade is likely to be more complicated.
Furthermore, the upgrade should only be merged in the CI pipeline reports success.
If this MR has failed, we can spend a little time investigating to see if a trivial upgrade could achieve compatiblity to the new library.
But significant upgrade efforts should not occur on this MR, as part of the release tagging process.
Instead, significant work should be scheduled for a subsequent milestone.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: 2842a845900c9d4629452f46c425e0a14ff3037c
Maven: 0.20.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | ------------------ | ------------------ |
| core-lib-azure | 0.19.0-rc8 | |
| core-lib-gcp | 0.19.0-rc3 | 0.19.0-rc3 |
| os-core-lib-aws | 0.16.1 | 0.16.1 |
| obm | 0.18.0 | 0.18.0 |
| oqm | 0.18.0 | 0.18.0 |
| os-core-common | 0.19.0-rc6, 0.18.0 | 0.16.0, 0.19.0-rc1 |
| os-core-lib-ibm | 0.17.0-rc4 | 0.16.0 |
| osm | 0.18.0 | 0.18.0 |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.1, 2.13.2.2 | 2.13.2.2 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.17.2, 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1 | 2.17.2, 2.13.3 |
| (3rd Party) org.springframework.spring-webflux | 5.3.24 | |
| (3rd Party) org.springframework.spring-webmvc | 5.3.22 | 5.3.22 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade-2
SHA: c9a5cdd17e2ed8e3f9de9bce74badbb5eee01898
Maven: 0.20.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | ---------------- | -------------- |
| core-lib-azure | 0.19.0 | |
| core-lib-gcp | 0.19.0 | 0.19.0 |
| os-core-lib-aws | 0.16.1 | 0.16.1 |
| obm | 0.19.0 | 0.19.0 |
| oqm | 0.19.0 | 0.19.0 |
| os-core-common | 0.19.0 | 0.16.0, 0.19.0 |
| os-core-lib-ibm | 0.17.0-rc4 | 0.16.0 |
| osm | 0.19.0 | 0.19.0 |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.1, 2.13.2.2 | 2.13.2.2 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.17.2, 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1 | 2.17.2, 2.13.3 |
| (3rd Party) org.springframework.spring-webflux | 5.3.24 | |
| (3rd Party) org.springframework.spring-webmvc | 5.3.22 | 5.3.22 |M16 - Release 0.19https://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/276Upgrade First Party Library Dependencies for Release 0.172022-10-04T05:12:42ZDavid Diederichd.diederich@opengroup.orgUpgrade First Party Library Dependencies for Release 0.17This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any...This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any library that is older than the previous release will be left as-is, since the upgrade is likely to be more complicated.
Furthermore, the upgrade should only be merged in the CI pipeline reports success.
If this MR has failed, we can spend a little time investigating to see if a trivial upgrade could achieve compatiblity to the new library.
But significant upgrade efforts should not occur on this MR, as part of the release tagging process.
Instead, significant work should be scheduled for a subsequent milestone.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: 8ac10f432bd53cc6e9c5147650a0fdf6caef8589
Maven: 0.17.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | -------- | ---------------------- |
| core-lib-azure | 0.16.0 | |
| core-lib-gcp | 0.16.0 | 0.3.25 |
| os-core-lib-aws | 0.16.1 | 0.16.1 |
| obm | 0.16.0 | |
| oqm | 0.16.0 | |
| os-core-common | 0.16.0 | 0.16.0, 0.3.18 |
| os-core-lib-ibm | 0.16.0 | 0.16.0 |
| osm | 0.16.0 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.13.2.2 | 2.13.2.2, 2.9.9.3 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.17.2, 2.13.3, 2.11.1 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1 | 2.17.2, 2.13.3, 2.11.2 |
| (3rd Party) org.springframework.spring-webflux | 5.3.12 | |
| (3rd Party) org.springframework.spring-webmvc | 5.3.22 | 5.3.22, 5.1.17.RELEASE |
```
Warning: Found Vulnerable Spring WebFlux dependency (<5.2.20 || >=5.3.0 <5.3.18)
└─ _Root_
└─ org.opengroup.osdu.partition-azure == 0.17.0-SNAPSHOT
└─ com.azure.spring.azure-spring-boot-starter-active-directory == 3.4.0
└─ org.springframework.boot.spring-boot-starter-webflux == 2.4.12
└─ org.springframework.spring-webflux == 5.3.12
```
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade
SHA: 2981c5d58c0bd60e07b66d0861ec5f1b3cc8ee16
Maven: 0.17.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | -------- | ---------------------- |
| core-lib-azure | 0.17.0 | |
| core-lib-gcp | 0.17.0 | 0.3.25 |
| os-core-lib-aws | 0.17.0 | 0.17.0 |
| obm | 0.17.0 | |
| oqm | 0.17.0 | |
| os-core-common | 0.17.0 | 0.17.0, 0.3.18 |
| os-core-lib-ibm | 0.17.0 | 0.17.0 |
| osm | 0.17.0 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.13.2.2 | 2.13.2.2, 2.9.9.3 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.17.2, 2.13.3, 2.11.1 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1 | 2.17.2, 2.13.3, 2.11.2 |
| (3rd Party) org.springframework.spring-webflux | 5.3.22 | |
| (3rd Party) org.springframework.spring-webmvc | 5.3.22 | 5.3.22, 5.1.17.RELEASE |M14 - Release 0.17https://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/225Upgrade core lib azure version to 0.16.0-rc22022-07-06T10:23:55ZAnubhav AronUpgrade core lib azure version to 0.16.0-rc2Feature Addition:
Destination for log either Geneva or ApplicationInsights is decided by already added Flag -DApplicationInsightsFeature Addition:
Destination for log either Geneva or ApplicationInsights is decided by already added Flag -DApplicationInsightsAnubhav AronAnubhav Aronhttps://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/161Updating partition service to use RedisAzureCache2023-02-04T05:29:54ZShiv SinghUpdating partition service to use RedisAzureCacheShiv SinghShiv Singhhttps://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/157Updating partition service to use RedisAzureCache2022-03-28T10:00:23ZShiv SinghUpdating partition service to use RedisAzureCacheShiv SinghShiv Singhhttps://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/88Updating OSDU dependencies2021-09-03T10:31:36ZDavid Diederichd.diederich@opengroup.orgUpdating OSDU dependenciesUpdating OSDU dependencies, to maintain use of the latest release among those that were previously doing soUpdating OSDU dependencies, to maintain use of the latest release among those that were previously doing soM8 - Release 0.11David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/226Updating Azure Core Lib Library Version to 0.16.0-rc22022-07-06T10:27:57ZAnubhav AronUpdating Azure Core Lib Library Version to 0.16.0-rc2Feature
Deciding destination of logs [Geneva or ApplicationInsights] using already added flag -DApplicationInsightsEnabledFeature
Deciding destination of logs [Geneva or ApplicationInsights] using already added flag -DApplicationInsightsEnabledAnubhav AronAnubhav Aronhttps://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/113Update pom.xml2021-11-08T05:45:57ZShrikant GargUpdate pom.xmlupdating ibm pomupdating ibm pomShrikant GargShrikant Garghttps://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/104Update .gitlab-ci.yml2021-10-12T12:33:26ZVladislav Shishko (EPAM)Update .gitlab-ci.ymlhttps://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/13Update error code response for integration testing2020-10-05T16:07:54ZDuvelis CaraoUpdate error code response for integration testingUpdated test cases response code when Token used to call the service is not valid or no token is sentUpdated test cases response code when Token used to call the service is not valid or no token is sentethiraj krishnamanaiduNeelesh ThakurAlok JoshiDuvelis CaraoSviatoslav Nekhaienkoethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/231Update devops/ibm/ibm-partition-deploy/files/configMap_update_script.sh2022-07-12T06:25:46ZKaliprasanna BasuUpdate devops/ibm/ibm-partition-deploy/files/configMap_update_script.shKaliprasanna BasuKaliprasanna Basuhttps://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/484Update deploy.yaml2023-11-21T11:40:45ZDanyl Sandler [EPAM / GCP]Update deploy.yamlM22 - Release 0.25https://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/267Update configMap_update_script.sh2023-05-13T05:03:40ZKaliprasanna BasuUpdate configMap_update_script.shAnuj GuptaAnuj Guptahttps://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/219update-cm config2022-06-21T07:12:43ZKaliprasanna Basuupdate-cm configKaliprasanna BasuKaliprasanna Basuhttps://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/87There are latencies (more than 300 seconds) on Partition API.2021-08-27T19:04:04ZDmitrii GerashchenkoThere are latencies (more than 300 seconds) on Partition API.https://community.opengroup.org/osdu/platform/system/partition/-/issues/16
An inspection showed that there is 2 minutes timeout for Azure TableStorage which can be the cause of the latencies.
10 minutes latency reproduced locally with ...https://community.opengroup.org/osdu/platform/system/partition/-/issues/16
An inspection showed that there is 2 minutes timeout for Azure TableStorage which can be the cause of the latencies.
10 minutes latency reproduced locally with the following conditions:
1. Endpoints GET /api/partition/v1/partitions or /api/partition/v1/partitions/{partitionId}
2. Not data in cache.
3. Azure Table storage is unavailable or responding too slow.
4. Many requests to API (more than 500).
Presumably, if a cache became outdated during high-load many simultaneous requests are send to TableStorage.
All requests which were sent before TableStorage response caching will create new requests to TableStorage and will be waiting for response up to 2 minutes. Finally, the API latency grows.
The solution is to use a cluster lock during the request to TableStorage. It's a copy of this solution from the Entitlements repository:
https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/blob/master/provider/entitlements-v2-azure/src/main/java/org/opengroup/osdu/entitlements/v2/azure/service/GroupCacheServiceAzure.java#L81
@Qualifier("cachedPartitionServiceImpl") was removed to make the bean "CachedPartitionServiceImpl" overridable.
CachedPartitionServiceImpl (defined in partition-core) was redefined with ProviderCachedPartitionServiceImpl (defined in partition-azure).
CachedPartitionService interface was introduced to resolve ambiguities for beans CachedPartitionService and PartitionServiceImpl. Both of them inherit IPartitionService. Now CachedPartitionService resolves ambiguities instead of @Qualifier("cachedPartitionServiceImpl").
New code was tested with the same conditions and the latency didn't grow.Dmitrii GerashchenkoDmitrii Gerashchenko