ADR: Policy enforcement in search

Policies cannot be applied in search as post-filter.

Status

  • Proposed
  • Trialing
  • Under review
  • Approved
  • Retired

Context & Scope

OSDU has adopted Rego as the language to define policies and Open Policy Agent as an internal solution to manage and enforce the policies. To enforce a policy, OSDU service calls policy services which internally calls OPA API

POST /v1/data/{policyId}
Content-Type: application/json`

with provided input for the policy decision.

This works very well if input data for evaluation is of low cardinality. It becomes a challenge if the size of input and output can be of very large size as it happens with search. In search, a user provides a query that determines the data (s)he is interested in. In addition to user query, OSDU platform must also apply all the policies before returning the result to the user. Doing this as two separate steps is not feasible because:

  • Aggregations (e.g., total count) would require fetching all documents that match user query (e.g., >100 millions), evaluate policies on that intermediate result and then calculate aggregation on top of result of policy evaluation.
  • Query with offset and query with cursor could not leverage Elasticsearch native functionality.

Optimal approach is to combine user query and policy evaluation in one operation issued against Elasticsearch index. Since Elasticsearch does not support Rego (and there is no roadmap to support it), policy would need to be available as Elasticsearch Query DSL. This can happen in one of two ways:

  1. Add the ability to "translate" policy written in Rego to Elasticsearch Query DSL
  2. Delegate the responsibility to the end user and support search policies written as Elasticsearch Query DSL

Tradeoff Analysis

Tradeoff Rego -> ES Query DSL translation approach ES Query DSL as policy language approach
Usability High Very low
Supported policy complexity Low High
Susceptibility to user errors Medium High
Performance impact Limited impact No impact

Decision

Expose an API that translates partially evaluated policies written in Rego to Elasticsearch Query DSL and append them to user provided query by leveraging OPA Elastic extensions.

Consequences

Out of the box, we will only support policies written using the following OPA Built-in Functions:

Comparison

  • ==
  • !=
  • <
  • <=
  • >
  • >=

Strings

  • contains

Regex

  • re_match

Support for other OPA functions that might be required by OSDU stakeholders would need to be prioritized and added separately.

Edited by Chad Leong