Partition issueshttps://community.opengroup.org/osdu/platform/system/partition/-/issues2021-10-01T11:44:27Zhttps://community.opengroup.org/osdu/platform/system/partition/-/issues/16Partition service's (azure-provider) latency is more than 300 seconds2021-10-01T11:44:27ZDmitrii GerashchenkoPartition service's (azure-provider) latency is more than 300 secondsThere are latencies (more than 300 seconds) on Partition API (azure-provider).
An inspection showed that there is 2 minutes timeout for Azure TableStorage which can be the cause of the latencies.
10 minutes latency reproduced locally w...There are latencies (more than 300 seconds) on Partition API (azure-provider).
An inspection showed that there is 2 minutes timeout for Azure TableStorage which can be the cause of the latencies.
10 minutes latency reproduced locally with the following conditions:
1. Endpoints GET /api/partition/v1/partitions or /api/partition/v1/partitions/{partitionId}
2. Not data in cache.
3. Azure Table storage is unavailable or responding too slow.
4. Many requests to API (more than 500).
Presumably, if a cache became outdated during high-load many simultaneous requests are send to TableStorage.
All requests which were sent before TableStorage response caching will create new requests to TableStorage and will be waiting for response up to 2 minutes. Finally, the API latency grows.
The solution is to use a cluster lock during the request to TableStorage. It's a copy of this solution from the Entitlements repository:
https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/blob/master/provider/entitlements-v2-azure/src/main/java/org/opengroup/osdu/entitlements/v2/azure/service/GroupCacheServiceAzure.java#L81
@Qualifier("cachedPartitionServiceImpl") was removed to make the bean "CachedPartitionServiceImpl" overridable.
CachedPartitionServiceImpl (defined in partition-core) was redefined with ProviderCachedPartitionServiceImpl (defined in partition-azure).
CachedPartitionService interface was introduced to resolve ambiguities for beans CachedPartitionService and PartitionServiceImpl. Both of them inherit IPartitionService. Now CachedPartitionService resolves ambiguities instead of @Qualifier("cachedPartitionServiceImpl").
New code was tested with the same conditions and the latency didn't grow.Dmitrii GerashchenkoDmitrii Gerashchenkohttps://community.opengroup.org/osdu/platform/system/partition/-/issues/15Upgrade Core GCP Dependency2022-02-11T21:56:28ZDavid Diederichd.diederich@opengroup.orgUpgrade Core GCP Dependencyhttps://community.opengroup.org/osdu/platform/system/partition/-/issues/14Upgrade Core IBM Dependency2022-02-11T21:56:46ZDavid Diederichd.diederich@opengroup.orgUpgrade Core IBM Dependencyhttps://community.opengroup.org/osdu/platform/system/partition/-/issues/13Upgrade Core Common Dependency2022-02-11T21:56:50ZDavid Diederichd.diederich@opengroup.orgUpgrade Core Common Dependencyhttps://community.opengroup.org/osdu/platform/system/partition/-/issues/12Convert read level operations: audit logs to debug logs2022-11-24T10:27:49ZPreksha Beohar-SlbConvert read level operations: audit logs to debug logsFor Partition Service : we want to convert all the application audit logs into the debug logs for all the Read operations.For Partition Service : we want to convert all the application audit logs into the debug logs for all the Read operations.Preksha Beohar-SlbPreksha Beohar-Slbhttps://community.opengroup.org/osdu/platform/system/partition/-/issues/11Upgrade dependencies to resolve security vulnerabilities2022-11-24T12:49:06ZRostislav Vatolinvatolinrp@gmail.comUpgrade dependencies to resolve security vulnerabilitiesPartition service has dependencies with critical bugs and vulnerabilities:
Updating:
guava, because it has security vulnerabilities: [CVE-2020-8908](https://nvd.nist.gov/vuln/detail/CVE-2020-8908)
bom for netty because of: [CVE-2021-2...Partition service has dependencies with critical bugs and vulnerabilities:
Updating:
guava, because it has security vulnerabilities: [CVE-2020-8908](https://nvd.nist.gov/vuln/detail/CVE-2020-8908)
bom for netty because of: [CVE-2021-21290](https://nvd.nist.gov/vuln/detail/CVE-2021-21290) and [CVE-2021-21409](https://nvd.nist.gov/vuln/detail/CVE-2021-21409) and [CVE-2021-21295](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-21295)
Upgrading version of spring: [WS-2016-7107](https://www.whitesourcesoftware.com/vulnerability-database/WS-2016-7107) and [WS-2020-0293](https://www.whitesourcesoftware.com/vulnerability-database/WS-2020-0293) and [CVE-2020-5421](https://nvd.nist.gov/vuln/detail/CVE-2020-5421)
Fixes were applied in common libraries.https://community.opengroup.org/osdu/platform/system/partition/-/issues/10Customized readiness check API2022-11-24T12:48:15ZMingyang ZhuCustomized readiness check APIWe'd like to use spring boot built-in actuator health endpoint for the partition service health check API, and implement a customized health indicator.
Partition service implements the cache layer, different cloud provider's implementat...We'd like to use spring boot built-in actuator health endpoint for the partition service health check API, and implement a customized health indicator.
Partition service implements the cache layer, different cloud provider's implementation implements the cache differently. It will be good to make sure the cache infrastructure and connection are ready before the pod serves the traffic.
To achieve this, the service will implement the custom health indicator, and get a dummy key from the cache instance. No matter it is memcache, redis, it expects no exception to claim the pod is ready to serve the traffic. This health check can be enabled or disabled by the configuration.Mingyang ZhuMingyang Zhuhttps://community.opengroup.org/osdu/platform/system/partition/-/issues/9Incorrect response for incorrect request2021-03-19T13:58:44ZRiabokon Stanislav(EPAM)[GCP]Incorrect response for incorrect request
POST https://<partition_service>/api/partition/v1/partitions/Test1-2571428 with body
```
{
"properties_invalid": {
}
}
```
AR: Response is 404 with body "Partition does not exist."
ER: 400 Bad Request, since "properties" fiel...
POST https://<partition_service>/api/partition/v1/partitions/Test1-2571428 with body
```
{
"properties_invalid": {
}
}
```
AR: Response is 404 with body "Partition does not exist."
ER: 400 Bad Request, since "properties" field should be mandatory.
I suppose we have to consider another implementation related to an annotation 'Valid' https://community.opengroup.org/osdu/platform/system/partition/-/blob/master/partition-core/src/main/java/org/opengroup/osdu/partition/api/PartitionApi.java#L50https://community.opengroup.org/osdu/platform/system/partition/-/issues/8Partition Service Authorization2022-10-09T07:51:09ZJasonPartition Service Authorization**Overview**: Currently, there is some ambiguity about how calls to partition service should be authorized. At a high level, there are two approaches being taken right now:
1. Authorizing all calls that come to partition service from a s...**Overview**: Currently, there is some ambiguity about how calls to partition service should be authorized. At a high level, there are two approaches being taken right now:
1. Authorizing all calls that come to partition service from a service account. Currently only Azure takes the approach where it authorizes all requests to partition service from a service principal.
2. Authorizing calls where the caller has the entitlements `service.partition.admin`. Currently IBM, GCP, and AWS seem to take this approach.
My understanding from reading the relevant issue in this repo [here](https://community.opengroup.org/osdu/platform/system/partition/-/issues/1) was that the Azure-style implementation was correct for partition service since it is an internal service that is needed before Entitlements is online in the system. Is this my misunderstanding? If not, could someone provide clarity about the correct approach to authorizing calls to partition service?
Reference:
[AWS Auth Code](https://community.opengroup.org/osdu/platform/system/partition/-/blob/master/provider/partition-aws/src/main/java/org/opengroup/osdu/partition/provider/aws/security/AuthorizationService.java)
[Azure Auth Code](https://community.opengroup.org/osdu/platform/system/partition/-/blob/master/provider/partition-azure/src/main/java/org/opengroup/osdu/partition/provider/azure/utils/AuthorizationService.java)
[IBM Auth Code](https://community.opengroup.org/osdu/platform/system/partition/-/blob/master/provider/partition-ibm/src/main/java/org/opengroup/osdu/partition/provider/ibm/security/AuthorizationService.java)
[GCP Auth Code](https://community.opengroup.org/osdu/platform/system/partition/-/blob/master/provider/partition-gcp/src/main/java/org/opengroup/osdu/partition/provider/gcp/security/AuthorizationService.java)JasonJasonhttps://community.opengroup.org/osdu/platform/system/partition/-/issues/7does Partition service APIs require a data-partition-id header?2020-12-23T11:20:53ZBhushan Radedoes Partition service APIs require a data-partition-id header?As per [documentation](https://community.opengroup.org/osdu/platform/system/partition/-/blob/master/docs/tutorial/Partition.md) all partition service APIs are not consuming any data-partition-id heders but the integration test module is ...As per [documentation](https://community.opengroup.org/osdu/platform/system/partition/-/blob/master/docs/tutorial/Partition.md) all partition service APIs are not consuming any data-partition-id heders but the integration test module is consuming data-partition-id headers for all Partition service APIs HTTP requests.
does Partition service APIs require a data-partition-id header?ethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/partition/-/issues/5Partition Service - New API to list all tenants2020-11-05T17:04:18ZDuvelis CaraoPartition Service - New API to list all tenants## Context & Scope
Partition service will provide a new API :
- List all partitions Id (GET)
## Use case
Legal service CRON (legalTag status change) and Schema service requires list of all partitions.
## Consequences
Client lib for P...## Context & Scope
Partition service will provide a new API :
- List all partitions Id (GET)
## Use case
Legal service CRON (legalTag status change) and Schema service requires list of all partitions.
## Consequences
Client lib for Partition service will be updated to support this new API.ethiraj krishnamanaiduDuvelis Caraoethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/partition/-/issues/4Modify contract to capture sensitive flag for partition specific secrets config2020-10-08T20:16:10ZNeelesh ThakurModify contract to capture sensitive flag for partition specific secrets configProblem: Partition secret configurations available via partition service can pose following security issue:
1. All Secrets are exposed by default to any service regardless if they need them or not.
2. Secrets are held in memory cache bo...Problem: Partition secret configurations available via partition service can pose following security issue:
1. All Secrets are exposed by default to any service regardless if they need them or not.
2. Secrets are held in memory cache both at the partition service and the service client library.
3. Potential for logging secret values to the central logger are increased due to secrets being sent in Microservice HTTP Response Objects.
a. Trace Logs are often used to dump http request and response objects between services for debugging purposes.
Solution: Provide a mechanism to distinguish secret and non-secret partition configuration and delegate responsibility of consuming secret using cloud native libraries at service level.
**Current**
```
public class PartitionInfo {
@Builder.Default
Map<String, Object> properties = new HashMap<>();
}
```
e.g.
```
{
"properties": {
"complianceRuleSet": "shared",
"storageAccountKey": "test-storage-**secret**"
}
}
```
**Proposed**
```
public class PartitionInfo {
@Builder.Default
Map<String, Property> properties = new HashMap<>();
}
public class Property {
@Builder.Default
private boolean sensitive = false;
private Object value;
}
```
e.g.
```
{
"properties": {
"complianceRuleSet": {
"sensitive": false,
"value": "shared"
},
"storageAccountKey": {
"sensitive": true,
"value": "test-storage-**key**"
}
}
}
```ethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/partition/-/issues/3ADR Support self-signed certificates for elasticsearch2020-09-10T18:00:18ZRiabokon Stanislav(EPAM)[GCP]ADR Support self-signed certificates for elasticsearch## Context and Scope
Search Service does not support HTTPS connections with self-signed certificates for elastic search.
## Decision
Add a property 'security.https.certificate.trust' into application-*.properties.
If it is 'true', we ...## Context and Scope
Search Service does not support HTTPS connections with self-signed certificates for elastic search.
## Decision
Add a property 'security.https.certificate.trust' into application-*.properties.
If it is 'true', we will use `TrustSelfSignedStrategy()` in `org.opengroup.osdu.search.util.ElasticClientHandler` that trusts self-signed certificates.
## Rational
We would use self-signed certificates for elastic search instead of signed certificates.
## Consequences
These changes could affect all providers due to these things will be implemented in search-core.Dmitriy RudkoDmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/partition/-/issues/2Partition Service Should Support Updating Existing Secrets in a Partition2020-12-02T21:49:32ZMatt WisePartition Service Should Support Updating Existing Secrets in a Partition## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The Partition Service currently supports 3 API Operations.
1. Create Partition (POST)
2. Get Partition (GET)
3. Delete Partition...## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The Partition Service currently supports 3 API Operations.
1. Create Partition (POST)
2. Get Partition (GET)
3. Delete Partition (DELETE)
Currently, the only way to add new secrets or update existing ones to the KV store for a partition is to delete it and recreate it. This leads to extra API calls and risks missing putting back some data that was not meant to be touched.
## Decision
If this is implemented, KV stores will be simpler to update existing values on without having to delete existing data.
## Rationale
Partitions need to be updated with new secrets and edits to existing ones. In order to minimize the complexity of this operation for external callers, having a route to support this will simplify partition editing processes.
## Consequences
New API route in partition has to be supported/tested by all implementors of the serviceM1 - Release 0.1ethiraj krishnamanaiduDania Kodeih (Microsoft)JoeChris ZhangRucha DeshpandeMatt Wiseethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/partition/-/issues/1Partition API routes are missing standard OSDU Roles2023-09-27T12:19:38ZMatt WisePartition API routes are missing standard OSDU RolesRoutes in the partition service use a custom authorization filter that does not use Roles, but rather hardcodes a specific account type 'DomainAdminServiceAccount'.
pre-auth block in API methods:
```java
@PreAuthorize("@authorizationFil...Routes in the partition service use a custom authorization filter that does not use Roles, but rather hardcodes a specific account type 'DomainAdminServiceAccount'.
pre-auth block in API methods:
```java
@PreAuthorize("@authorizationFilter.hasPermissions()")
```
Authorization filter has the following function:
```java
public boolean hasPermissions() {
return authorizationService.isDomainAdminServiceAccount();
}
```
**Should the partition service use standard roles rather than user types to be compliant with other services?**
Other services use the following:
```java
@PreAuthorize("@authorizationFilter.hasRole('" + DeliveryRole.VIEWER + "')")
```M1 - Release 0.1ethiraj krishnamanaiduJoeMatt Wiseethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/partition/-/issues/6Partition Service to Support Multiple Data Partitions2021-02-22T21:00:13ZAn NgoPartition Service to Support Multiple Data Partitions**Overview**
As part of the effort to enable multiple data partitions in OSDU, there is a need for a Partition Service that is responsible for creating and retrieving the partition specific properties on behalf of calling services. The ...**Overview**
As part of the effort to enable multiple data partitions in OSDU, there is a need for a Partition Service that is responsible for creating and retrieving the partition specific properties on behalf of calling services. The service will encapsulate the data currently held in the secrets store and the "tenantinfo" datastore. Encapsulation in this case is referring to isolation via the service interface vs. any implication of where or how the secrets data, in particular, is physically stored.<br><br>
The Partition Service will be the means of decoupling services from the logic of partition creation/info retrieval/deletion and will be fairly generic, i.e. essentially a means to store and retrieve key/value pair information relevant to data partitions. As a service rather than a client library, the Partition Service also provides a logical point to implement features related to performance and scalability. Additionally, the Partition Service will be language independent and available to all services without a separate implementation for each language family.
**Details**
For the Azure implementation of Partition Service, the following should hold:
+ This will be a service that can only be accessed by any other services within the (multi-cluster) service mesh.
+ It will have a 5-minute TTL on GET data partition info responses
+ It will allow dirty reads if TTL has expired but can’t be updated by the client
+ This should be implemented to the same standards as other OSDU services (technology stack, SPIs etc.) but with an Azure implementation first. This forces the interface of the API (partitionInfo) to be more generic.
+ All data will be stored in Azure Key Vault whether this is secret or not. This means no partition data in CosmosDb.
+ Service principal credentials are needed to access create and delete APIs. Get API should only be accessible within the cluster and has no public access outside the cluster.M1 - Release 0.12021-02-26