README.md

Policy Service

Policy service is an OSDU service used to manage (view, create, update, delete) and evaluate dynamic policies. Dynamic policies are written using Rego language. Rego queries are assertions on data. It is a declarative language so it is convenient for writing policies.

Policy service APIs are consistent with other OSDU APIs in a way that they require Bearer token as authorization header and data partition as data-partition-id header for all the calls. Similarly, user making the call needs to be in a necessary service group to be authorized to make the call. For policy service, the relevant service groups are service.policy.user and service.policy.admin (configurable in conf.py but this will not update inline doc for swagger, etc).

Recently policy service was migrated from Flask to FastAPI, this was done in part because:

• Data validation (via Pydantic), serialization and deserialization (for building an API)
• Blazing performance over Flask
• Async request capability
• Automatic documentation (via JSON Schema and OpenAPI)

Prerequisites for testing or running Policy Service:

• OPA The Open Policy Agent (OPA, pronounced “oh-pa”) is an open source, general-purpose policy engine that unifies policy enforcement across the stack. OPA provides a high-level declarative language that lets you specify policy as code and simple APIs to offload policy decision-making from your software.

  • Min. version 0.44 is recommended
  • OPA must be configured to poll for new policies in [example init.yaml)(tests/gc_init.yaml).
  • For testing purposes a low polling is probably best.
  • OPA must also be bootstrapped with an initial bundle for instance and data partition. example
  • Be sure environment variable OPA_URL points to the OPA service endpoint. It defaults to http://localhost:8181 for testing purposes. Some of the options for starting OPA include:
  • opa run --server
  • make opa to run OPA in docker container
  • make debugopa start OPA in debug mode running in docker container
  • or run in Kubernetes, etc.
  • OPA container needs the following environment variables are required to make evaluations work:
  • LEGAL_BASE_URL
  • ENTITLEMENTS_BASE_URL

• Entitlement Service

  • Entitlement service is assumed to be not available for unit tests, however it is required to support integration tests. With headers Authorizations (Bearer Token) and data_partition_id, this service is used to determine which groups the request is associated with. The user must be a part of conf.USER_PERMISION (service.policy.user by default) or conf.ADMIN_PERMISION (service.policy.admin by default) for that particular OSDU data partition or testing will fail. Entitlement can also be bypassed for testing purposes by enabling conf.MOCK_ENTITLEMENT

• Legal Service

  • Legal service is needed to run some integration tests (directly to get legal tags and indirectly when expecting OPA to talk to legal service). However legal is not directly used by the policy-service. However policies should make OPA call the legal service.

• CLOUD_PROVIDER

  • used for all Cloud Service Providers. Used to initialize storage client for given CSP. This storage client will be used to manipulate the bundle file content. If not set a value of "MOCK" will be assumed, this will mock using a CSP for unit testing. MOCK is never to be used in production.
  • A value of LOCAL will bypass using CSP and attempt to update OPA directly va OPA_URL. This is great for local development, testing, tiny environments (that don't have multiple OPA Pods), on-premise or for unsupported cloud environments.

  Currently supported values of CLOUD_PROVIDER:

  • baremetal
  • aws
  • azure
  • gcp
  • ibm
  • LOCAL
  • MOCK or undefined

  Related Settings/Environmental variables:

  • POLICY_BUCKET - used by AWS, Google Cloud and IBM to determine which bucket is used for providing bundle files. Service identity running policy service needs to have write permission to contents in this bucket. Note us-east-1 is only region supported at this time for AWS.
  • CONTAINER_NAME - used by Azure to determine which container is used for providing bundle files. Service principal running policy service needs to have write permission to contents in this bucket.
  • STORAGE_ACCOUNT - used by Azure to determine which account is used for providing bundle files. Service principal running policy service needs to have write permission to contents in this bucket.
  • ENDPOINT_URL, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY - used by IBM (yes IBM). Please note only region us-east-1 is currently supported.
  • MINIO_ENDPOINT, MINIO_SECRET_KEY, MINIO_ACCESS_KEY - used by Baremetal (Reference Architecture).

• Limit Number of Groups for Performance Reasons:

  • The number of groups should be configured to what is necessary. Performance will degrade when there are many groups. This is especially true for translation API which is used by search. Ideally less than 25 groups should be used. However more groups will work, but you may have performance problems as this number of groups grows. At some point the number of groups will break the request payload size however and policy service will return a 413 or 400 error for translate API requests. This configuration error may break search. This can be seen when the number of groups is 1000+.

• Useful software but not required:

  • Make - GNU make utility
  • ttab - programmatically open a new terminal tab
  • envsubst - GNU gettext utilities

• Port - the default for the policy service is 8080. In Kubernetes you can still expose this to port 80.

Testing Policy Service

The test directory contains pytest unit and integration tests and their associated data files.

• Unit tests assume no services are available.
• Many of the test scripts assume your running on a Mac, Linux or inside Docker
• Integration tests assume:

• Open Policy Agent (OPA),
• Policy Service (app server) are running. Settings/Environment variables that impact testing:
  • LOG_LEVEL if not set will be "INFO"
  • LOG_LEVEL_TRANSLATE if not set will be set to the same as LOG_LEVEL
  • CLOUD_PROVIDER
    • POLICY_BUCKET - if CLOUD_PROVIDER is set to Google Cloud or IBM
  • conf.ENTITLEMENTS_BASE_URL ENTITLEMENTS_BASE_URL
  • conf.LEGAL_BASE_URL LEGAL_BASE_URL
  • OPA_URL for example (http://opahost:port)
  • TOKEN with admin privileges (or cmd line option --token) - Should just contain only the token, not contain "Bearer " export TOKEN="xyz". Token should be valid for data partition and not expired.
  • DATA_PARTITION (or cmd line option --data_partition)
  • conf.USE_BUNDLES USE_BUNDLES - This is now the default an asssumed to enabled
  • conf.ENABLE_DEV_DIAGNOSTICS ENABLE_DEV_DIAGNOSTICS - turns on additional /diag API methods. Do not enable these in production environments, however they are extremely useful in Sandbox, development and supporting CI/Integration Testing. This will also cause eval API to create some "dump" json files, so the policy service will need local write access to current working directory.
  • ENABLE_ADMIN_UI with this turned on /adminui/ URL will be turned on within the container, which serves up /assets. This is added during build phase. When testing locally and not using containers it would be best to just run npm start
  • --service_url if you want to connect to a policy service other than default
• NOTE: all environment variables and permissions for running policy service should also be exactly duplicated when running pytest. The pytest will also run a local copy of the policy service for some tests in addition to testing at service_url.
  • Caching is now enabled for responses from OPA. This is configurable in conf.py

Running tests:

To start tests you might want to consider using the following from parent directory:

• make localtest to execute both unit and integration tests locally.

• settings can be overridden on make command line, for example make localtest PORT=80
• Alternatively uvicorn main:app --port 8080 can be run from the app directory or uvicorn app.main:app --port 8080 can be run from the root directory

Developer Notes:

• Python 3.9 is expected. Some headers are not python 3.10 compatible.
• pip install -r requirements.txt
• In a future release this requirements list will be trimmed down for production purposes.
• Alternatively you can run use the Docker container.
• make gcp_test_suite will launch configure OPA bundles, deploy OPA, launch policy-service and start automated unit and integration testing. This assumes a Mac with all optional software is installed.

Developer Google Cloud Platform (GC) Notes:

• If testing with user credentials, you might need to run gcloud auth application-default login and gsutil config.
• If you're using a service account with a json file, gcloud auth activate-service-account --key-file=auth.json and gsutil config -e
• You may also need to set SA_FILE_PATH and/or GOOGLE_APPLICATION_CREDENTIALS env variables to the path of the json file.

Admin UI & CLI:

• frontend/adminui contains the Angular code for AdminUI. This should be considered highly experimental and not turned on for production systems.
• To start the Admin UI:

cd ../frontend/adminui
npm start

• frontend/admincli is the admin CLI for policy-service. Use of this easy to use full featured client is strongly encouraged. For more information on Admin CLI.

Translate:

For more information on translate API.

Building Docker Containers:

• Currently docker container is based upon Python 3.9 slim buster

• make build or docker build --network host -t policy-service:latest .

• The make build command will also convert the AdminUI into static files to frontend/adminui/dist/policyservice this will get added during the docker build phase.

• make run for running the policy-service in Docker. Keep in mind that environment variables are still required for the policy-service even running in Docker. make run handles a lot of that for you.

• Alternatively you could do something like this to run policy service in docker:

   docker run  -it --rm \
        -e OPA_URL=http://host.docker.internal:8181 \
        -e ENTITLEMENTS_BASE_URL=https://yourentitlementsservice \
        -e ENTITLEMENTS_BASE_PATH=/api/entitlements/v2/groups \
        -e LEGAL_BASE_URL=https://yourlegalservice \
        -e TOKEN="your bearer token here" \
        -e ENABLE_DEV_DIAGNOSTICS=1 \
        -e ENABLE_ADMIN_UI=1 \
        -e GOOGLE_CLOUD_PROJECT=your_google_project \
        -e CLOUD_PROVIDER=gcp \
        -e POLICY_BUCKET=your_policy_bucket_in_google_cloud \
        -v $HOME/.config/gcloud:/root/.config/gcloud \
        --name policy-service -p 8080:8080 policy-service:latest

Todo / Planned work:

• Admin UI

APIs

OpenAPI Note: All APIs except /health require authentication. /health was left out so it can be used in readiness checks.

Bundles

Types of bundles and bundle bootstraping

Two types on bundles are bootstraped: instance and partition bundles. Instance bundle cannot be changed through policy service but only by directly c hanging the instance bundle file in the bundle server. Partition bundle can be changed via policy service.

All bundles (instance and all partition bundles) need to be bootstraped in bundle server. Naming convention for these bundles is bundle.tar.gz for instance bundle and bundle-data-partition-id.tar.gz for each partition.

Instance bundle contains .manifest with following contents:

{
    "roots": ["osdu/instance"]
}

Instance policies should be osdu/instance/xyz.rego. No extra / in the path. Strange results may occur if addition paths are added.

Partition bundle contains .manifest with following contents:

{
    "roots": ["osdu/partiton/data_partition_id"]
}

Partition policies should be osdu/partition/<data_partition_id>/xyz.rego. No extra / in the path. Strange results may occur if addition paths are added.

To define a policy within a data partition, one must include a namespace that is specific for the given data partition. For example, dataauthz poli cy for data partition data_partition_id can look like this:

package osdu.partition["dp1-100"].dataauthz

import data.osdu.instance.dataauthz

records := data.osdu.instance.dataauthz.records

Examples of the above can be found in tests directory.

Supported OSDU Services

The following services can be configured (some by default in M14) to use Policy Service:

• Search this requires the search.rego policy in your data partition to work.

• Storage this may require the storage.rego policy in your data partition to work.