Policy issueshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues2024-02-26T16:37:13Zhttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/96ADR: Make OPA configuration dynamic updatable2024-02-26T16:37:13ZShane HutchinsADR: Make OPA configuration dynamic updatable## Status
- [x] Proposed
- [ ] Trialing
- [ ] Under review
- [x] Approved
- [ ] Retired
## Context
OSDU has adopted Rego as the language to define policies and [Open Policy Agent](https://www.openpolicyagent.org/docs/latest/) as an int...## Status
- [x] Proposed
- [ ] Trialing
- [ ] Under review
- [x] Approved
- [ ] Retired
## Context
OSDU has adopted Rego as the language to define policies and [Open Policy Agent](https://www.openpolicyagent.org/docs/latest/) as an internal solution to manage and enforce the policies. To enforce a policy, various OSDU services call policy service which internally calls OPA API. Some services (storage) bypass policy service and make low level calls to OPA directly.
Today OPA configuration is strictly managed by CSPs, generally with a [kubernetes config map](https://kubernetes.io/docs/concepts/configuration/configmap/). By having this static and only updatable with backend it breaks the ability to add a partition with [partition](https://community.opengroup.org/osdu/platform/system/partition) create API.
As a result, once a new partition is the following services are become impacted:
- Storage
- Search
Any services that depends on the above, including but not limited to:
- Indexer
- [seismic-dms-suite seismic-store-service v4](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/tree/master/app/sdms-v4)
For additional context see the following issues and links:
- https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/94
- [Support Multi Partition Policies in OPA](https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/wikis/Support-Multi-Partition-Policies-in-OPA)
The workaround:
- Workaround requires backend access and manual updates for updating the OPA configuration. See [workaround](https://osdu.pages.opengroup.org/platform/security-and-compliance/policy/bundles/#adding-a-new-partition-to-osdu)
## Scope
Implement APIs to manage OPA configuration.
## Solution
Update the Policy Service /bootstrap API to also create, update and manage the configmap for OPA.
![image](/uploads/d7b7a0791ef1afb1897a067abdc0996f/image.png)
## Consequences
- Kubernetes permissions to allow read and update of OPA config map (opa-agent) will be required.
- CSPs will need to not update the config map once created.
## Futures
- At a later date partition service could be configured to call policy bootstrap API to remove the burden of having to call an additional API.M23 - Release 0.26Shane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/95Feature Request - Need capability to write policy based on data records prope...2024-03-15T15:38:30ZDadong ZhouFeature Request - Need capability to write policy based on data records propertiesFrom Fabrice HAÜY [SLB] on Slack:
Hi Team, I'm looking for some updated information / roadmap, as from our latest conversations at the OSDU F2F in London, I understood that currently, the policy engine only knowns about id, kind, legal ...From Fabrice HAÜY [SLB] on Slack:
Hi Team, I'm looking for some updated information / roadmap, as from our latest conversations at the OSDU F2F in London, I understood that currently, the policy engine only knowns about id, kind, legal tag, and acl, making it not possible to create policy entitlements based on the value of a property of the record. I'm looking for information surrounding this limitation and when it'll be unlocked. thank you in advance
cc @hmarkovic @hutchins @chadhttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/127Add "/readiness" endpoint2024-02-19T16:10:03ZYan Sushchynski (EPAM)Add "/readiness" endpointHaving a distinct endpoint for readiness probes, signifying service readiness for traffic reception, would greatly enhance operational efficiency. In our scenario, this endpoint could invoke the readiness endpoints of both OPA and Entitl...Having a distinct endpoint for readiness probes, signifying service readiness for traffic reception, would greatly enhance operational efficiency. In our scenario, this endpoint could invoke the readiness endpoints of both OPA and Entitlements services. This process ensures all Policy dependencies are ready, affirming Policy readiness.
More info about readiness can be found here: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/Shane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/116Add a sample DataAuthz policy using LegalTag extension properties for policy ...2023-10-31T22:24:02ZDadong ZhouAdd a sample DataAuthz policy using LegalTag extension properties for policy integration testingAdd a sample DataAuthz policy using LegalTag extension properties which will be included in policy integration testing.Add a sample DataAuthz policy using LegalTag extension properties which will be included in policy integration testing.Dadong ZhouDadong Zhouhttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/104Azure Monitor - Policy service logs not found in Azure App Insights2024-01-03T15:29:57ZKelly ZhouAzure Monitor - Policy service logs not found in Azure App InsightsHi,
We found that we can't find any policy service logs in Azure App Insight, is that by design or are we missing any configuration? we wonder if the monitoring of policy service in Azure deployment going well and how OSDU community ma...Hi,
We found that we can't find any policy service logs in Azure App Insight, is that by design or are we missing any configuration? we wonder if the monitoring of policy service in Azure deployment going well and how OSDU community managed it. Any response will be much appreciated.
@Srinivasan_Narayanan @nursheikh
Thank you!Shane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/102Static code analysis for python libraries2023-10-31T22:06:01ZShane HutchinsStatic code analysis for python librariesWe should adopt this community ADR in policy service pipeline:
https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/15#note_227939
This would help devs identify and fix static analysis code findingsWe should adopt this community ADR in policy service pipeline:
https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/15#note_227939
This would help devs identify and fix static analysis code findingshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/94OPA Breaks Adding a New Partition to OSDU2023-12-26T14:20:13ZShane HutchinsOPA Breaks Adding a New Partition to OSDUCurrently there is only a manual workaround to add a new partition to OSDU https://osdu.pages.opengroup.org/platform/security-and-compliance/policy/bundles/#adding-a-new-partition-to-osdu
Currently if you do not follow these manual step...Currently there is only a manual workaround to add a new partition to OSDU https://osdu.pages.opengroup.org/platform/security-and-compliance/policy/bundles/#adding-a-new-partition-to-osdu
Currently if you do not follow these manual steps. There will never be a bundle for the partition and Policy Service will error on all requests for that partition. OPA configuration (which generally comes from a kubernetes config map) isn't updated to know to attempt to read bundle for that partition.
This is known to break Policy, Storage, Search and Seismic DMS (seismic-store-service v4).
Impacts Milestone releases: M14-M18M20 - Release 0.23Hrvoje MarkovicNeelesh ThakurRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comDadong ZhouYauhen Shaliou [EPAM/GCP]Shane HutchinsSrinivasan NarayananYong Zengvikas ranaHrvoje Markovichttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/92Update Policy service to support latest python2024-03-15T15:40:16ZShane HutchinsUpdate Policy service to support latest pythonPolicy Service requires Python 3.9.x.
Update Policy service to use a more recent version of Python.
Created from https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/91Policy Service requires Python 3.9.x.
Update Policy service to use a more recent version of Python.
Created from https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/91M24 - Release 0.27Shane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/79Add Open Telemetry (OTEL) to Policy Service2023-10-31T22:06:15ZShane HutchinsAdd Open Telemetry (OTEL) to Policy ServiceAdd Open Telemetry (OTEL) to Policy Service
- Focus on Trace/Span support
In a later issue add metric and logs supportAdd Open Telemetry (OTEL) to Policy Service
- Focus on Trace/Span support
In a later issue add metric and logs supportShane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/63Policy Bundle implementation - race condition bug2023-10-31T22:27:21ZShane HutchinsPolicy Bundle implementation - race condition bugWhile updates to policies are not expected to happen frequently, there is a potential race condition in the put/delete bundle code if parallel update/delete jobs to the same data partition. This race condition exists if the requests come...While updates to policies are not expected to happen frequently, there is a potential race condition in the put/delete bundle code if parallel update/delete jobs to the same data partition. This race condition exists if the requests come into the same policy service (and handled via async) or between multiple pods/containers running the policy service.
DoD:
* Develop design the corrects issue and still can work with bundles (or local).
* Present design to team for approval.
* Create follow-on ticket for implementation of solution of agreed upon solution.https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/54User manual for dynamic policies2023-10-31T22:07:43ZHrvoje MarkovicUser manual for dynamic policiesDadong ZhouDadong Zhouhttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/52Rego to Elastic DSL translation performance evaluation2023-10-30T14:51:38ZHrvoje MarkovicRego to Elastic DSL translation performance evaluationImplement a POC to determine if Rego to Elastic DSL translation is feasible to ensure all policies are written in Rego.
Based on outcome:
Alternative 1 (satisfactory performance): Change search to call OPA directly and perform partial...Implement a POC to determine if Rego to Elastic DSL translation is feasible to ensure all policies are written in Rego.
Based on outcome:
Alternative 1 (satisfactory performance): Change search to call OPA directly and perform partial evaluation and translation
Alternative 2 (unsatisfactory performance): Start supporting policy for search written in Elastic Query DSL
Related to #22.https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/31Large scale end to end system test2022-09-28T19:44:03ZHrvoje MarkovicLarge scale end to end system testLoad > 1M records to ensure dynamic policies scale system wide.Load > 1M records to ensure dynamic policies scale system wide.Hrvoje MarkovicHrvoje Markovic