Policies: scalability and performance

Status

• Proposed
• Trialing
• Under review
• Approved
• Retired

Context & Scope

When evaluating a policy engine, its scalability and performance are one of the critical trade-offs. We should seek to test the example policies with data volumes and complexities that represent what is considered 'normal' user scenarios.

Decision

We expect to support following volumes in an OSDU R3 data partition:
< 1e9 data items < 1000 of policies

We expect following performance: < 100 ms for policy evaluation when a typical request is made to storage or search in case of typical OSDU deployment.

Rationale

We need to prove that not only does it work in the sense of enforcing policies but also that the performance is acceptable in normal usage.

Consequences

Likely to be some consequences in terms of the number and complexity of policies that can be supported. Expect limitations on the granularity at which entitlement can be applied (without impacting performance)

When to revisit

Once criteria in the decision is not met.

Decision criteria and trade-offs

Granularity at which entitlement can be set versus performance Number and complexity of policies versus performance

Decision timeline

July 2020