ADR: Policy enforcement in search
Policies cannot be applied in search as post-filter.
Status
-
Proposed -
Trialing -
Under review -
Approved -
Retired
Context & Scope
OSDU has adopted Rego as the language to define policies and Open Policy Agent as an internal solution to manage and enforce the policies. To enforce a policy, OSDU service calls policy services which internally calls OPA API
POST /v1/data/{policyId}
Content-Type: application/json`
with provided input for the policy decision.
This works very well if input data for evaluation is of low cardinality. It becomes a challenge if the size of input and output can be of very large size as it happens with search. In search, a user provides a query that determines the data (s)he is interested in. In addition to user query, OSDU platform must also apply all the policies before returning the result to the user. Doing this as two separate steps is not feasible because:
- Aggregations (e.g., total count) would require fetching all documents that match user query (e.g., >100 millions), evaluate policies on that intermediate result and then calculate aggregation on top of result of policy evaluation.
- Query with offset and query with cursor could not leverage Elasticsearch native functionality.
Optimal approach is to combine user query and policy evaluation in one operation issued against Elasticsearch index. Since Elasticsearch does not support Rego (and there is no roadmap to support it), policy would need to be available as Elasticsearch Query DSL. This can happen in one of two ways:
- Add the ability to "translate" policy written in Rego to Elasticsearch Query DSL
- Delegate the responsibility to the end user and support search policies written as Elasticsearch Query DSL
Tradeoff Analysis
Tradeoff | Rego -> ES Query DSL translation approach | ES Query DSL as policy language approach |
---|---|---|
Usability | High | Very low |
Supported policy complexity | Low | High |
Susceptibility to user errors | Medium | High |
Performance impact | Limited impact | No impact |
Decision
Expose an API that translates partially evaluated policies written in Rego to Elasticsearch Query DSL and append them to user provided query by leveraging OPA Elastic extensions.
Consequences
Out of the box, we will only support policies written using the following OPA Built-in Functions:
Comparison
-
== -
!= -
< -
<= -
> -
>=
Strings
-
contains
Regex
-
re_match
Support for other OPA functions that might be required by OSDU stakeholders would need to be prioritized and added separately.