# Policy Service
Policy service is an OSDU service used to manage (view, create, update, delete) and evaluate dynamic policies. Dynamic policies are written using [Rego](https://www.openpolicyagent.org/docs/latest/policy-language/) language. Rego queries are assertions on data. It is a declarative language so it is convenient for writing policies.

Policy service APIs are consistent with other OSDU APIs in a way that they require Bearer token as authorization header and data partition as data-partition-id header for all the calls. Similarly, user making the call needs to be in a necessary service group to be authorized to make the call. For policy service, the relevant service groups are service.policy.user and service.policy.admin (configurable in [conf.py](./conf.py) but this will not update inline doc for swagger, etc). 

Recently policy service was migrated from Flask to FastAPI, this was done in part because:
* Data validation (via Pydantic), serialization and deserialization (for building an API)
* Blazing performance over Flask
* Async request capability
* Automatic documentation (via JSON Schema and OpenAPI)

## Prerequisites for testing or running Policy Service:
* OPA
    The [Open Policy Agent](https://www.openpolicyagent.org) (OPA, pronounced “oh-pa”) is an open source, general-purpose policy engine that unifies policy enforcement across the stack. OPA provides a high-level declarative language that lets you specify policy as code and simple APIs to offload policy decision-making from your software. 
    - Min. version 0.44 is recommended
    - OPA must be configured to poll for new policies in [example init.yaml)(tests/gc_init.yaml).
    - For testing purposes a low polling is probably best.
    - OPA must also be bootstrapped with an initial bundle for instance and data partition. [example](tests/gc_bootstrap_policy.sh)
    - Be sure environment variable `OPA_URL` points to the OPA service endpoint. It defaults to `http://localhost:8181` for testing purposes.
    Some of the options for starting OPA include:
    * `opa run --server`
    * `make opa` to run OPA in docker container
    * `make debugopa` start OPA in debug mode running in docker container
    * or run in Kubernetes, etc.
    - OPA container needs the following environment variables are required to make evaluations work:
    * LEGAL_BASE_URL
    * ENTITLEMENTS_BASE_URL 
* Entitlement Service
    * Entitlement service is assumed to be not available for unit tests, however it is required to support integration tests. With headers 
    Authorizations (Bearer Token) and data_partition_id, this service is used to determine which groups the request is associated with.
    The user must be a part of `conf.USER_PERMISION` (service.policy.user by default) or `conf.ADMIN_PERMISION` (service.policy.admin by default) for that particular OSDU data partition or testing will fail. Entitlement can also be bypassed for testing purposes by enabling `conf.MOCK_ENTITLEMENT`
* Legal Service
    * Legal service is needed to run some integration tests (directly to get legal tags and indirectly when expecting OPA to talk to legal service). However legal is not directly used by the policy-service. However policies should make OPA call the legal service.
* CLOUD_PROVIDER
    - used for all Cloud Service Providers. Used to initialize storage client for given CSP. This storage client will be used to manipulate the bundle file content. If not set a value of "MOCK" will be assumed, this will mock using a CSP for unit testing. MOCK is never to be used in production.
    - A value of LOCAL will bypass using CSP and attempt to update OPA directly va `OPA_URL`. This is great for local development, testing, tiny environments (that don't have multiple OPA Pods), on-premise or for unsupported cloud environments.

    Currently supported values of `CLOUD_PROVIDER`:
    * baremetal
    * aws
    * azure
    * gcp
    * ibm
    * LOCAL
    * MOCK or undefined

    Related Settings/Environmental variables:
    - `POLICY_BUCKET` - used by AWS, Google Cloud and IBM to determine which bucket is used for providing bundle files. Service identity running policy service needs to have write permission to contents in this bucket. Note us-east-1 is only region supported at this time for AWS.
    - `CONTAINER_NAME` - used by Azure to determine which container is used for providing bundle files. Service principal running policy service needs to have write permission to contents in this bucket.
    - `STORAGE_ACCOUNT` - used by Azure to determine which account is used for providing bundle files. Service principal running policy service needs to have write permission to contents in this bucket.
    - `ENDPOINT_URL`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` - used by IBM (yes IBM). Please note only region `us-east-1` is currently supported.
    - `MINIO_ENDPOINT`, `MINIO_SECRET_KEY`, `MINIO_ACCESS_KEY` - used by Baremetal (Reference Architecture).
* Limit Number of Groups for Performance Reasons:
    * The number of groups should be configured to what is necessary. Performance will degrade when there are many groups. This is especially true for translation API which is used by search. Ideally less than 25 groups should be used. However more groups will work, but you may have performance problems as this number of groups grows. At some point the number of groups will break the request payload size however and policy service will return a 413 or 400 error for translate API requests. This configuration error may break search. This can be seen when the number of groups is 1000+.

* Useful software but not required:
    * Make - [GNU make utility](https://www.gnu.org/software/make/)
    * ttab - [programmatically open a new terminal tab](https://www.npmjs.com/package/ttab)
    * envsubst - [GNU gettext utilities](https://www.gnu.org/software/gettext/manual/html_node/envsubst-Invocation.html)

* Port - the default for the policy service is 8080. In Kubernetes you can still expose this to port 80.

## Testing Policy Service
The [test directory](./test) contains pytest unit and integration tests and their associated data files.
* Unit tests assume no services are available.
* Many of the test scripts assume your running on a Mac, Linux or inside Docker
* Integration tests assume:
- Open Policy Agent (OPA),
- Policy Service (app server) are running. Settings/Environment variables that impact testing:
    * `LOG_LEVEL` if not set will be "INFO"
    * `LOG_LEVEL_TRANSLATE` if not set will be set to the same as LOG_LEVEL
    * CLOUD_PROVIDER
        * `POLICY_BUCKET` - if `CLOUD_PROVIDER` is set to Google Cloud or IBM
    * conf.ENTITLEMENTS_BASE_URL `ENTITLEMENTS_BASE_URL`
	* conf.LEGAL_BASE_URL `LEGAL_BASE_URL`
	* `OPA_URL` for example (http://opahost:port)
    * TOKEN with admin privileges (or cmd line option --token) - Should just contain only the token, not contain "Bearer "
        `export TOKEN="xyz"`. Token should be valid for data partition and not expired.
    * `DATA_PARTITION` (or cmd line option --data_partition)
    * conf.USE_BUNDLES `USE_BUNDLES` - This is now the default an asssumed to enabled
    * conf.ENABLE_DEV_DIAGNOSTICS `ENABLE_DEV_DIAGNOSTICS` - turns on additional /diag API methods. Do not enable these in production environments, however they are extremely useful in Sandbox, development and supporting CI/Integration Testing. This will also cause eval API to create some "dump" json files, so the policy service will need local write access to current working directory.
    * `ENABLE_ADMIN_UI` with this turned on /adminui/ URL will be turned on within the container, which serves up /assets. This is added during build phase. When testing locally and not using containers it would be best to just run `npm start`
    * --service_url if you want to connect to a policy service other than default
- NOTE: all environment variables and permissions for running policy service should also be exactly duplicated when running pytest. The pytest will also run a local copy of the policy service for some tests in addition to testing at service_url.
    * Caching is now enabled for responses from OPA. This is configurable in conf.py

## Running tests:
To start tests you might want to consider using the following from parent directory:
* `make localtest` to execute both unit and integration tests locally. 
- settings can be overridden on make command line, for example `make localtest PORT=80`
- Alternatively `uvicorn main:app --port 8080` can be run from the app directory or `uvicorn app.main:app --port 8080` can be run from the root directory

## Developer Notes:
- Python 3.9 is expected. Some headers are not python 3.10 compatible.
- `pip install -r requirements.txt`
- In a future release this requirements list will be trimmed down for production purposes.
- Alternatively you can run use the Docker container.
- `make gcp_test_suite` will launch configure OPA bundles, deploy OPA, launch policy-service and start automated unit and integration testing. This assumes a Mac with all optional software is installed.

### Developer Google Cloud Platform (GC) Notes:
- If testing with user credentials, you might need to run `gcloud auth application-default login` and `gsutil config`.
- If you're using a service account with a json file, `gcloud auth activate-service-account --key-file=auth.json` and `gsutil config -e`
- You may also need to set SA_FILE_PATH and/or GOOGLE_APPLICATION_CREDENTIALS env variables to the path of the json file.

### Admin UI & CLI:
* frontend/adminui contains the Angular code for AdminUI. This should be considered highly experimental and not turned on for production systems.
* To start the Admin UI:
```
cd ../frontend/adminui
npm start
```
* frontend/admincli is the admin CLI for policy-service. Use of this easy to use full featured client is strongly encouraged.
For more information on [Admin CLI](../frontend/admincli/README.md).

## Translate:
For more information on [translate API](./translate/README.md).

## Building Docker Containers:
* Currently docker container is based upon Python 3.9 slim buster
* `make build` or `docker build --network host -t policy-service:latest .`
* The `make build` command will also convert the AdminUI into static files to frontend/adminui/dist/policyservice this will get added during the docker build phase.

* `make run` for running the policy-service in Docker. Keep in mind that environment variables are still required for the policy-service even running in Docker. `make run` handles a lot of that for you.
* Alternatively you could do something like this to run policy service in docker:
```
   docker run  -it --rm \
        -e OPA_URL=http://host.docker.internal:8181 \
        -e ENTITLEMENTS_BASE_URL=https://yourentitlementsservice \
        -e ENTITLEMENTS_BASE_PATH=/api/entitlements/v2/groups \
        -e LEGAL_BASE_URL=https://yourlegalservice \
        -e TOKEN="your bearer token here" \
        -e ENABLE_DEV_DIAGNOSTICS=1 \
        -e ENABLE_ADMIN_UI=1 \
        -e GOOGLE_CLOUD_PROJECT=your_google_project \
        -e CLOUD_PROVIDER=gcp \
        -e POLICY_BUCKET=your_policy_bucket_in_google_cloud \
        -v $HOME/.config/gcloud:/root/.config/gcloud \
        --name policy-service -p 8080:8080 policy-service:latest
```

## Todo / Planned work:
* Admin UI

## APIs
[OpenAPI](../docs/openapi.yaml)
Note: All APIs except /health require authentication. /health was left out so it can be used in readiness checks.

## Bundles
### Types of bundles and bundle bootstraping
Two types on bundles are bootstraped: instance and partition bundles. Instance bundle cannot be changed through policy service but only by directly c
hanging the instance bundle file in the bundle server. Partition bundle can be changed via policy service.

All bundles (instance and all partition bundles) need to be bootstraped in bundle server. Naming convention for these bundles is _bundle.tar.gz_ for
instance bundle and _bundle-data-partition-id.tar.gz_ for each partition.

### Instance bundle contains _.manifest_ with following contents:
```
{
    "roots": ["osdu/instance"]
}
```
Instance policies should be osdu/instance/xyz.rego. No extra / in the path. Strange results may occur if addition paths are added.

### Partition bundle contains _.manifest_ with following contents:
```
{
    "roots": ["osdu/partiton/data_partition_id"]
}
```
Partition policies should be osdu/partition/<data_partition_id>/xyz.rego. No extra / in the path. Strange results may occur if addition paths are added.

To define a policy within a data partition, one must include a namespace that is specific for the given data partition. For example, _dataauthz_ poli
cy for data partition _data_partition_id_ can look like this:
```
package osdu.partition["dp1-100"].dataauthz

import data.osdu.instance.dataauthz

records := data.osdu.instance.dataauthz.records
```
Examples of the above can be found in tests directory.

## Supported OSDU Services
The following services can be configured (some by default in M14) to use Policy Service:
* [Search](https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/docs/tutorial/PolicyService-Integration.md) this requires the search.rego policy in your data partition to work.

* [Storage](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/PolicyService-Integration.md) this may require the storage.rego policy in your data partition to work.