Partition service

Status

• Proposed
• Trialing
• Under review
• Approved
• Retired

Context & Scope

The OSDU data platform supports the concept of data partitions and the availability of multiple partitions for a single deployment.

A data partition provides the highest level of data isolation within a single OSDU deployment. All access rights are governed at a partition level and data is separated in a way that allows for the partitions life cycle and deployment to be handled independently from one another.

A data partition helps solve many concerns

• Data Access – More sensitive data can be deployed to a separate partition providing an isolated access boundary separate to the rest of their data providing higher security
• Data Management – Organizations can distribute partitions to different segments/units for self-management where appropriate for reasons of budgeting, access rights, rights of use, licensing etc.
• Joint Ventures - Partitions can be set up for joint ventures between different organizations
• Data Residency – An organization can deploy a partition to a location with unique residency/sovereignty concerns to the rest of their data. Data partitions are an orthogonal concern to the region/location of a deployment but can be used to help simplify concerns in cases of data sovereignty and residency concerns.
• Development - Organizations want to develop applications on top of their deployments. Data partitions allow the test data used in development to be kept completely separate from their real data stopping pollution scenarios in their workflows.
• SaaS delivery - ability to deliver different partitions for different customers. Supporting encryption of data in different partitions with customer specific keys for additional data protection.

Most of the APIs in OSDU expose a custom header 'data-partition-id'. The value of this points to the data partition the client is requesting access to.

In the current implementation the specific properties that relate to a partition are largely captured through the shared ‘TenantInfo’ data object. Other properties specific to a partition e.g. secrets are often stored in different key vaults or databases and also need to be retrieved.

These are then high-frequency usage points in the system and we want to rationalize these concerns in a central service to help maintain the integrity of the system.

Decision

For R3 we will encapsulate the storage and retrieval of partition specific properties within its own service.

Rationale

This will encapsulate the storage and retrieval of partition specific information. This will decouple implementations away from shared databases improving maintainability.

We can also encapsulate the performance, reliability, and scaling needs behind this service. Retrieving partition information is a high frequency, single point of failure in the system so having each service duplicate this increases the risk of service availability.

Consequences

Deployments will need to switch to store the partition specific properties via the service.

Services will need to be switched to retrieve partition specific properties via the service.

These changes can happen gradually overtime and should be specific to each provider. One providers consumption of this new service should not affect another providers implementation.

Trade-off Analysis - Input to decision

An alternative approach is the current Java shared code implementation. Here we see that a large amount of the partition specific properties are retrieved in a single POJO called ‘TenantInfo’. Other partition specific properties like Elastic Search credentials are retrieved from secret stores. There may be other variations on where this data may be retrieved from.

All of these are single points of failure in the system that are retrieved on every request. If you cannot get these properties, the request cannot be completed for any service that supports the data-partition-id header.

Therefore, each service must implement the logic to retrieve this information or at least each language/framework in OSDU needs a shared component for the most common properties.

These properties are needed on every request and having multiple implementations make a performant, elastic and reliable implementation both less likely and harder to achieve.

Similarly coupling all services to specific databases in the system makes maintainability worse as there is tighter coupling of the system so any changes at the data layer end up requiring cascading changes in multiple services.

However, these implementations do already exist and are working today. There is an effort involved in both creating this service and then migrating the other services to use this instead.

Decision criteria and trade-offs

• Elasticity
• Performance
• Reliability
• Maintainability
• Cost of change