Home issueshttps://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues2020-07-14T15:38:45Zhttps://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/22Cloud Native Security2020-07-14T15:38:45ZStephen Whitley (Invited Expert)Cloud Native Security# Cloud Native Security
Use cloud native resource isolation and security when possible
The value of a OSDU depends on its ability to collect and contain data from a variety of users, departments and organizations/companies. To encourage...# Cloud Native Security
Use cloud native resource isolation and security when possible
The value of a OSDU depends on its ability to collect and contain data from a variety of users, departments and organizations/companies. To encourage contribution, the OSDU must earn the trust of its participants.
## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [X] Approved
- [ ] Retired
## Context
## Decision
OSDU will take advantage of the resource isolation and access control mechanism of the cloud provider rather than attempting to implement these itself. It will take advantage of resource isolation (Projects/Subscriptions/VPCs) between companies to ensure complete isolation and use Access Control Lists based on groups or roles within a company for finer grain control.
## Rationale
Leveraging the native security controls of the cloud
1. avoids introducing complexity/vulnerabilities in the data ecosystem implementation,
2. reduces the requirement for privileged accounts and
3. avoids the possibility of someone gaining access to data by circumventing the OSDU services.
## Consequences
1. The finest granularity of access control is dictated by the capabilities of the cloud provider
https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/21Policy Based Entitlements2020-07-02T11:29:56ZStephen Whitley (Invited Expert)Policy Based EntitlementsWe are often left to address the gaps from architectural principles (which stay at a pretty high and abstract level) to the actual implementation detail. Here is an attempt to bridge that gap by providing a set of Lightweight Architectur...We are often left to address the gaps from architectural principles (which stay at a pretty high and abstract level) to the actual implementation detail. Here is an attempt to bridge that gap by providing a set of Lightweight Architecture Decision Records (LADRs) which are simple to follow and can be implemented in a given team/project by the developers
# Decision Title
## Status
- [X] Proposed
- [X] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
- **Policies**: Contextual grants of permission (often legal: contractual, residency, trade compliance) hese are captured in [Entitlements User Stories](/OSDU-(C)/Architecture-Exploration-Topics/OSDU-R3-Architecture/Entitlements-User-Stories)
- **Access Control**: Explicit grants of permissions
- **Identity**: Consider related questions around [Authentication](/OSDU-(C)/Architecture-Exploration-Topics/OSDU-R3-Architecture/Authentication)
### Scale of Problem
- For a single, large multi-national enterprise scale company
- Duration: 10's of years
- Users: 10,000's of users working within potentially 1,000's of organizational scopes
- Content: 100's of millions of business objects with new types being introduced in a continuous basis.
## Decision
* We will support complex policies that are evaluated dynamically.
* Work needs to be done to collect examples of these policies
* We will review how to make these policies “expressible” from the perspective of an admin. The language of policy engines is powerful; but complex policies can look complex.
* We could use some help on the usability study because a LOT of policies will be unmanageable unless we think about the UI and how to organize them.
* We will look for an Open Source policy engine and replace the bespoke one that is in OpenDES.
* The evaluation will look at Usability and Expressiveness to make sure it meets the needs
* The evaluation will look at Performance and Latency to figure out
* where in the architecture / services it should be called
* where we need to collect the contextual information (user, location, contract, …) so that this does not become a bottleneck.
* We will not remove the OpenDES support for storage level ACLs
* Doing so would break every deployment of the data ecosystem and force reloading customer data we don’t have access to.
* We will continue to support Storage level ACLS for existing Deployments and for OSDU customers who wish additional security at the storage level
* If companies don’t want to use ACLs as additional security, then they can use a single service account for all queries and do all the entitlement management through policy.
## Rationale
## Consequences
## When to revisit
---
# Tradeoff Analysis - Input to decision
## Alternatives and implications
There are two broad classes of entitlements: precalculated and dynamically evaluated.
**Precalculated entitlements** are fast to evaluate and require the least context. A good example of a precalculated entitlement is assigning a role to an object and then matching a user's membership in that role. The attachment of the role to the object and the assignment of the user to the role happen in advance of determining access. So at the time of access, a simple matching process can determine with almost 100% accuracy, in more-or-less constant time whether the access should be granted or not. Speed and accuracy are the obvious advantages. The primary disadvantage is that the mechanism for determining who belongs in which group and why is left as an exercise for the operator. It is not obvious how an operator translates its needs for policy, compliance, business roles, or need-to-know into a set of roles/groups. The tools for organising users / groups / roles, etc. live outside OSDU.
**Dynamic evaluations** encode the rights of users in descriptive policies or rules. At the time of access, the relevant set of rules is determined and evaluated. This requires translating the operator's requirements for policy, compliance, business roles, need-to-know, etc. into a machine-readable policy language/set of rules. The primary advantage is the flexibility to describe access controls generally, even in advance of data loading. E.g., data with attribute X cannot be read by users with attribute Y. The primary disadvantage is that the more flexibility that is possible in the rules, the slower the system is likely to perform. The tools for writing rules that implement operators' needs will be inside OSDU, but such tools have to be created for OSDU.
The three proposals below generally differ by how much of these two approaches are used. To more deeply understand these three approaches, look at **[three worked entitlement examples](/OSDU-\(C\)/Architecture-Exploration-Topics/OSDU-R3-Architecture/WorkedEntitlementExamples)**.
#### Policies and ACLs
Many systems separate design and implementation between contextual grants and explicit access permissions. There are multiple reasons for this and they include: historical (requirements for contextual grants come later), different risk exposure, different performance vs freshness trade-offs, incompatible use cases, etc.
The policy/ACL approach is a hybrid approach applying precalculated access control assignments (e.g. owner, rights, permissions) with dynamically-calculated policy entitlements. Access decisions that can be decided purely by precalculated rights are decided that way. This means that in some situations, only ACLs are considered, whereas in other situations policy/compliance requirements *and* ACLs are considered.
Depending on the requirements, contextual grants can require dynamic rule evaluation or be implemented as separate consumption zones/patterns. There are different ways to determine whether policy/requirement rules are evaluated. It can be per service/API. E.g., calling service X, API Y requires policy evaluation, but calling service X API Z will succeed/fail based only on ACLs. Another mechanism is through consumption zones/patterns. E.g., when exporting the data to certain environments, (e.g., non-OSDU environments) policy evaluations might need to occur. Whereas exporting data to other environments (where OSDU entitlements are actively enforced) perhaps only ACLs are checked.
#### Everything as policy
With *everything as policy* all access decisions are dynamically evaluated. No distinction is made between contextual policies and the ones where the context is not important. Context is always considered in every access request. A uniform set of rules is evaluated at each time of access.
In this way policies can be implemented independent of specific data or specific users. The access control policies can resemble the original license terms, regulatory rules, or organizational policies that the operator adheres to.
#### Constrained policies
The constrained policy approach attempts to define a generic entitlements model that can express grants in form of rules, but make it usable in a big data space by limiting the impact on performance and cost. In this approach, rules are fundamentally dynamic, but the expressiveness of the rules / patterns / language for defining policies is constrained to a very small set. The goal is to achieve many of the benefits dynamic entitlement while limiting complexity. The limited complexity should allow a more performant system implementation.
**Possible constraints**:
- Whereas the everything-as-policy approach would regulate every service and every API one possible simplification would be to only entitle the service level. E.g., users can have all APIs on a service or none. Or, more likely, any regulation of what a user can do inside a service is left to the service itself to implement. This simplifies and speeds up the policy engine evaluation by pushing complex evaluations to individual OSDU services.
- Limited operations: define simply 3 levels of entitlement for all services: read, write, owner. Force all services to align their activities along these levels. Another similar set of operations would be CRUD. Force all OSDU services to entitle along those 4 operations.
- Lack of wildcards / patterns: some of the most time-consuming operations are string operations matching against patterns liks `*@example.com` or `urn:foo:bar:baz/*`. One way to constrain policy would be to forbid patterns and force enumeration of exact user and resource names.
## Decision criteria and tradeoffs
### Trade-off Points
- Granularity of authorization
- The end-user perspective (what the user wants)
- The system perspective (what the system inherently provides)
- Basic security concerns
- Does it require services with elevated privileges?
- Water-tightness of approach - how easy is it to circumvent?
- Does it satisfy common security requirements around basic operations like CRUD
- Performance
- How long to decide a given access request?
- Time it takes for a policy or an access control change to propagate through the system
- Scalability to thousands of users, thousands of groups, trillions of objects, thousands of policies
- Scalability/synchronizability across multiple installations in disparate geographic regions
- Usability
- Ability to manage permissions at scale
- Ability to understand / audit who actually has access under what conditions
- User-friendliness of the approach
- Susceptibility to user errors
- Technical feasibility and maintainability
- Can it be implemented
- Is it portable across cloud providers
- Is it portable across technologies
- Does it follow a known/existing standard?
- Are these existing technical solutions that we can leverage?
- How much leverages cloud native, established frameworks versus how much is purpose built by OSDU?
- Cost of Change
- Impact of migrating existing data currently in deployed instances
- Likelihood that approach would lead to future migrations of data
- Likelihood that the approach can be amended / adjusted without provoking substantial change in the future.
- Correctness
- Likelihood that the entitlement decision is "correct" at the moment it is made
- Ability to describe and enforce the majority of all business, legal, and regulatory requirements. Easy things are easy, exceptional things look exceptional.
## Decision timeline
Hrvoje MarkovicHrvoje Markovichttps://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/4Operator-controlled TLS Options2020-06-19T16:32:46ZPaco Hope (AWS)Operator-controlled TLS OptionsAs an operator, I can provide the public key certificate and corresponding private key for all TLS endpoints that handle my data. This covers externally-facing API endpoints. Anything that is deployed as part of deploying the OSDU data p...As an operator, I can provide the public key certificate and corresponding private key for all TLS endpoints that handle my data. This covers externally-facing API endpoints. Anything that is deployed as part of deploying the OSDU data platform (not the cloud provider's own endpoints).
1. Issuing the TLS certificate for the API endpoints from a CA / PKI that the operator controls.
* This must be importable into OSDU
* This must be renewable when it expires
2. The TLS cipher policy must be controlled to restrict connections to operator-approved TLS ciphers.
### Operator Inputs
- **Chevron**: This is mandatory for Chevron.
- **Repsol**: This is mandatory for Repsol.
- **Equinor**: We do have an internal PKI so we need to be able to configure trust between internal resources and the OSDU install. (*Paco comment: this might require BYOK for certificates, it might not*)
## Definition of Done
* An operator can provide one or more TLS certificates and they will be deployed to the externally-facing endpoints.
* An operator can indicate which valid TLS ciphers are acceptable/supported for their OSDU endpoints. (e.g., TLS 1.2 only)
* When connecting to the API endpoints, the operator's provided TLS certificate is presented in the TLS handshake.
* When connecting to the API endpoints, TLS connections are rejected unless the client selects an acceptable cipher.
* The requirements for an operator-provided X.509 certificate (e.g., signature, key size, etc) is documented so that operators can supply compatible certificates. Linking to the appopriate cloud provider's documentation will be necessary, but not sufficient.
* The requirements for an operator-provided TLS cipher list is documented so operators can select their cipher suites. Linking to the appropriate cloud provider's documentation for mechanisms will be necessary, but not sufficient.M1 - Release 0.1https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/5OSDU and platform logging requirements2021-04-22T21:28:09ZPaco Hope (AWS)OSDU and platform logging requirementsAll OSDU logs need to go to a specific, well-known location. That means:
### Logs must exist
* OSDU service logs (e.g., load service, search service, etc.)
* application platform logs (e.g., kubernetes, tomcat, nginx, apache, whatever)...All OSDU logs need to go to a specific, well-known location. That means:
### Logs must exist
* OSDU service logs (e.g., load service, search service, etc.)
* application platform logs (e.g., kubernetes, tomcat, nginx, apache, whatever)
* operating system level logs for VMs
* cloud service provider logs
### Logs must be protected
Logs go one of two ways:
1. They leave OSDU and go to an operator-provided location. In that case, security and management of logs is the operator's responsibility.
2. They remain in an OSDU-specific location (e.g., a log server, an S3 bucket, a cloud-native log aggregation service). In that case additional security requirements apply.
- Logs must be encrypted at rest
- Logs must be protected from unauthorised modification
- Logs must be protected from unauthorised access
- **ConocoPhillips** identified RBAC for log access
### Log Locations
- **Chevron**: Azure Log Analytics
- **Total**: Azure Monitor
- **Petronas**: LogRhythm
- **ConocoPhillips**: Splunk
- **Equinor**: Azure EventHub
### Log Retention
- **BP** highlighted data retention as a security concern. Logs are the one place where the platform itself produces data. Do we activate some automatic cloud-native log deletion?
- **Repsol** Log integrity measures are mandatory as well as a retention period of 13 months of the logs.
### Definition of Done
1. Log formats need to be documented and defined for each service and component at each of these levels (OSDU, app, OS, cloud).
2. Log locations are documented for each cloud provider choice.
3. The operator can indicate their preference for logs to either remain in an OSDU-specific location or be exported to another system.M1 - Release 0.1ethiraj krishnamanaiduRaj KannanJoeethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/6OSDU API Logging Requirements2020-07-14T20:51:29ZPaco Hope (AWS)OSDU API Logging RequirementsFor all OSDU APIs, we need to log a number of attributes in a standard format.
Each log entry should contain:
* The principal from the `id_token`
* Whether the invocation was allowed/denied
* What resources were involved
* Any API hea...For all OSDU APIs, we need to log a number of attributes in a standard format.
Each log entry should contain:
* The principal from the `id_token`
* Whether the invocation was allowed/denied
* What resources were involved
* Any API header containing sensitive information such as authorization token should not be logged
* Unambiguous date and time. Preferably in [ISO-8601](https://www.iso.org/iso-8601-date-and-time-format.html) format
* Origin IP address for caller
* Other information as appropriate to the service being invoked
All OSDU APIs will generate API invocation logs in JSON format. The OSDU data platform will not filter any of the output that is logged. Operators can redact, delete, or retain logs as required.
### Operator Input
* **Chevron** wants JSON format and imported into Azure Log Analytics.
* **ConocoPhillips** wants [CIM (Common Information Model)](https://docs.splunk.com/Documentation/CIM/4.15.0/User/Overview) format to be supported.
* **Repsol**: any Syslog, CEF, GELF, etc. is suitable.
* **Equinor**: CEF and JSON formats are acceptable.M1 - Release 0.1Raj KannanJoeRaj Kannanhttps://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/12Kubernetes Security Requirements2020-06-24T20:30:44ZPaco Hope (AWS)Kubernetes Security Requirements### Input from Operators
These requirements provided by
- **Chevron**
- **Total**
- **Repsol**
- **Shell**
### Requirements
- K8s diagnostic settings must be enabled and forwarded to Log Analytics
- Azure AD should be enabled in Kuber...### Input from Operators
These requirements provided by
- **Chevron**
- **Total**
- **Repsol**
- **Shell**
### Requirements
- K8s diagnostic settings must be enabled and forwarded to Log Analytics
- Azure AD should be enabled in Kubernetes Service
- Cluster RBAC must be enabled in Kubernetes Service
- Do not directly or indirectly grant cluster admin level access to developers
- The latest version of Kubernetes should be used . Autoupdate daemon to be used
- Ensure containers listen only on allowed ports
- Do not allow privileged containers in AKS
- Container build file must be auditable/visible
- Allow Secrets Injection into containersM1 - Release 0.1https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/13Static Code Analysis during CI/CD Pipeline2021-06-28T17:21:55ZPaco Hope (AWS)Static Code Analysis during CI/CD PipelineThis probably belongs in the [CI-CD pipelines](https://community.opengroup.org/osdu/platform/ci-cd-pipelines) board.
### Requirements
- Java code needs to pass through a static code analysis tool.
- Infrastructure code needs to pass th...This probably belongs in the [CI-CD pipelines](https://community.opengroup.org/osdu/platform/ci-cd-pipelines) board.
### Requirements
- Java code needs to pass through a static code analysis tool.
- Infrastructure code needs to pass through a static code analysis tool.
- Third party / open source code libraries that are used need to be scanned for vulnerabilities and for compatible open source licenses.
### Operator Input
* **Chevron** lists this as a requirement
* **Shell** lists this as a requirement
* **ExxonMobil** lists this as a requirement
* **BP** lists this as a requirement
### Definition of Done
- Code is scanned as a standard set of tests during the standard pipeline build
- Security tests failing can cause the build to failM1 - Release 0.1https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/17Network Security Requirements2020-09-22T03:40:20ZPaco Hope (AWS)Network Security RequirementsThe infrastructure that runs the OSDU Data platform must have limited outbound connectivity to the Internet.
How this is accomplished will vary by cloud provider. But it is important that individual containers, services, and hosts cann...The infrastructure that runs the OSDU Data platform must have limited outbound connectivity to the Internet.
How this is accomplished will vary by cloud provider. But it is important that individual containers, services, and hosts cannot make arbitrary connections to arbitrary outside IP addresses on arbitrary ports.
# Network security controls
## Egress
What network egress requirements are there for OSDU Data Platform services? E.g., if the data platform needs to/wants to connect to an outside data source, what network security requirements does the Operator impose?
Example controls include:
- Whitelisted domains and/or IP ranges.
- Proxying outbound HTTPS requests through an Operator-supplied proxy.
- For example, if the Load Service calls the Search Service, can that HTTPS API call pass through an HTTPS proxy.
- Egress to the Operator's on-premises network via a dedicated VPN channel
- Some operators want to block all outbound connection to internet IPs
## Ingress
Connections inbound are usually monitored.
- What level of logging / information is required for inbound connections?
## Transit
What network security controls are required inside the internal cloud network (e.g., between services)?
### Operator Input
- **Wintershall Dea**: And the Server(s) will have restrictions to access the Internet, at least it will be strictly controlled. HTTPS outbound only connections are possible. 2) The Deployment run in our on-premises Datacenter(s), here we have only Proxy-based Internet-Access for Servers. HTTPS outbound should be also no problem when it’s agreed.
- **ExxonMobil**: It is a requirement to have any outbound HTTPS pass through a proxy.
- **Shell**: WAF or L7 security inbound is required.https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/18Code Integrity, Access Control, Change Management2020-06-26T21:16:14ZPaco Hope (AWS)Code Integrity, Access Control, Change ManagementOperators want to know who has access/authority to commit code to the OSDU Data Platform. This doesn't need to be complex or long. But they need to know who can commit and how those permissions are handled. The process for accepting pull...Operators want to know who has access/authority to commit code to the OSDU Data Platform. This doesn't need to be complex or long. But they need to know who can commit and how those permissions are handled. The process for accepting pull requests/commits and making changes to the code should also be documented.
### Operator Inputs
- **Shell** lists this as a requirement.
### Definition of Done
Documentation and/or a link is probably sufficient. E.g., a link to the current list of approved committers, and a statement like "The OSDU OMC approves committers based on X, Y, Z". A statement about which workstream has authority over change management will satisfy that requirement, too.M1 - Release 0.1Raj KannanJoeRaj Kannanhttps://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/19Selectable External Content Encryption2020-06-19T14:07:41ZPaco Hope (AWS)Selectable External Content EncryptionSome sensitive content must be configured to be encrypted using an external encryption key source. Data that has the attribute **[TBD:ExternalEncryption]** must be encrypted prior to storage in the underlying cloud provider's storage sys...Some sensitive content must be configured to be encrypted using an external encryption key source. Data that has the attribute **[TBD:ExternalEncryption]** must be encrypted prior to storage in the underlying cloud provider's storage system. The encryption key for this encryption is pre-established by the operator. The external key providing system is reached via API call from the OSDU data platform service prior to encrypting/decrypting. This call can fail (because the operator has withdrawn consent to decrypt this data in the data platform), thus failures must be handled gracefully.
### Operator Input
- **ExxonMobil** lists this as a requirement to store sensitive data (e.g., annotations, commentary) in the OSDU data platform.
### Example: Export data
1. End user requests sensitive data to be exported
2. Data platform service retrieves encrypted data from cloud platform storage
3. Data platform service contacts external key provider to retrieve data key (that decrypts this data element)
4. Choice:
a. External key service replies negative: no key found / available. Data platform returns an error code.
b. External key service replies with a data key: Data platform decrypts the data and continues normally
### Example: Load data
1. End user requests sensitive data to be loaded. Manifest sets the **[TBD:ExternalEncryption]** in the manifest.
2. Data platform service contacts external key provider to retrieve **new** data key (that will encrypts this data element)
3. Choice:
a. External key service replies negative: no key available. Data platform returns an error code for this file.
b. External key service replies with a data key: Data platform encrypts the data with the provided data key and continues normallyhttps://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/20Don't preclude Endpoint Detection & Response2020-06-19T14:29:48ZPaco Hope (AWS)Don't preclude Endpoint Detection & ResponseWhere instances / VMs are deployed, do not preclude the use of operator-provided endpoint detection & response software installed on the nodes.
### Operator Inputs
- **Shell** lists this as a requirementWhere instances / VMs are deployed, do not preclude the use of operator-provided endpoint detection & response software installed on the nodes.
### Operator Inputs
- **Shell** lists this as a requirementM1 - Release 0.1https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/25Entitlement Inheritance2020-09-22T03:40:17ZStephen Whitley (Invited Expert)Entitlement Inheritance## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The OSDU data platform supports data derivatives as part of enrichment steps. These derivatives can be created from multiple p...## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The OSDU data platform supports data derivatives as part of enrichment steps. These derivatives can be created from multiple parent sources; each with a different entitlement model.
``` mermaid
graph LR
P[Enrichment]
S1((Source 1))
S2((Source 2))
S3((Source 3))
style S1 fill:#f9f,stroke:#333,stroke-width:2px
style S2 fill:#9ff,stroke:#333,stroke-width:2px
style S3 fill:#ff9,stroke:#333,stroke-width:2px
R((Result))
S1 --> P
S2 --> P
S3 --> P
P --> R
```
The entitlement of the Result can be a function of the Entitlements of each of the sources, the enrichment that was performed and the organization that created it.
## Decision
For R3 we will explicitly define the entitlements for the Result instead of trying to compute it from its source(s) or traversal of its source(s).
## Rationale
This will ensure that entitlement and performance of evaluation is deterministic.
## Consequences
Derivative data from multiple sources will have to be tagged explicitly.
## When to revisit
After R3 when we have experience and better understand the implications on
- usability
- performance
- maintainability
# Trade-off Analysis - Input to decision
We could try to compute the entitlements based on lineage; however, there is no guarantee that the information required to do this properly would be both available and complete.
This places the burden on establishing access rights for derivative data on the producer of this derivative. This producer would then explicitly access to the new data to other interested parties.
## Alternatives and implications
- **Lineage traversal**: Looking at all ancestor data in real time to evaluate right of access. This would rely on having all the information required to calculate access rights correctly and would have performance implications.
- **Calculated Entitlement**: This would avoid the performance issues of the above; but would still require access to all the information required to assess access rights; including from the producer of the derivative.
## Decision criteria and trade-offs
- Correctness
- Determinism
- Usability
- Performance
## Decision timeline
July 2020M1 - Release 0.1Hrvoje MarkovicHrvoje Markovichttps://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/32Policies: scalability and performance2021-07-20T19:54:08ZHrvoje MarkovicPolicies: scalability and performance## Status
- [X] Proposed
- [x] Trialing
- [x] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
When evaluating a policy engine, its scalability and performance are one of the critical trade-offs. We should seek to test th...## Status
- [X] Proposed
- [x] Trialing
- [x] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
When evaluating a policy engine, its scalability and performance are one of the critical trade-offs. We should seek to test the example policies with data volumes and complexities that represent what is considered 'normal' user scenarios.
## Decision
We expect to support following volumes in an OSDU R3 data partition:
< 1e9 data items
< 1000 of policies
We expect following performance:
< 100 ms for policy evaluation when a typical request is made to storage or search in case of typical OSDU deployment.
## Rationale
We need to prove that not only does it work in the sense of enforcing policies but also that the performance is acceptable in normal usage.
## Consequences
Likely to be some consequences in terms of the number and complexity of policies that can be supported.
Expect limitations on the granularity at which entitlement can be applied (without impacting performance)
## When to revisit
Once criteria in the decision is not met.
## Decision criteria and trade-offs
Granularity at which entitlement can be set versus performance
Number and complexity of policies versus performance
## Decision timeline
July 2020Hrvoje MarkovicHrvoje Markovic