NOTICE and documentation update

NOTICE

@@ -97,7 +97,7 @@ The following software have components provided under the terms of this license:
 - charset-normalizer (from https://github.com/Ousret/charset_normalizer)
 - coloredlogs (from https://coloredlogs.readthedocs.io)
 - exceptiongroup (from https://pypi.org/project/exceptiongroup/1.0.1/, https://pypi.org/project/exceptiongroup/1.1.1/)
-- fastapi (from https://pypi.org/project/fastapi/0.86.0/, https://pypi.org/project/fastapi/0.95.1/)
+- fastapi (from https://pypi.org/project/fastapi/0.86.0/, https://pypi.org/project/fastapi/0.95.2/)
 - h11
 - humanfriendly (from https://humanfriendly.readthedocs.io)
 - iniconfig (from http://github.com/RonnyPfannschmidt/iniconfig, https://pypi.org/project/iniconfig/2.0.0/)

docs/docs/api.md

 # API Documentation
 
-**OpenAPI Documentation**: <a href="https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/blob/master/docs/openapi.yaml" target="_blank"> https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/blob/master/docs/openapi.yaml</a>
+Policy service APIs are consistent with other OSDU APIs in a way that they require **Bearer token** as authorization header and **data partition** as _data-partition-id_ header for all the calls. Similarly, user making the call needs to be in a **necessary service group** to be authorized to make the call. For policy service, the relevant service groups are _service.policy.user_ and _service.policy.admin_.
+
+All the API's can be found **OpenAPI Documentation**: <a href="https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/blob/master/docs/openapi.yaml" target="_blank"> https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/blob/master/docs/openapi.yaml</a>
+
+## Policy APIs for Managing Policies
+
+### GET /api/policy/v1/policies
+
+Retrieves all the policies. This API gives the list of all the defined policies and it includes the policy definitions in the raw Rego form. It performs authorization check. The user making the call needs to be either _service.policy.user_ or _service.policy.admin_ in the provided data partition.
+
+<details>
+
+```
+curl -X 'GET' \
+  '/api/policy/v1/policies' \
+  -H 'accept: application/json' \
+  -H 'data-partition-id: osdu' \
+  -H 'correlation-id: 123' \
+  -H 'Authorization: Bearer <TOKEN>'
+```
+
+</details>
+
+### PUT /api/policy/v1/policies/osdu/partition/{data_partition}/{policy_id}
+
+Create or update a policy with given policy_id. This API will create/update policy definition with provided Rego expression and assign it with provided id. It performs authorization check. The user making the call needs to be _service.policy.admin_ in the provided data partition.
+
+<details>
+
+```
+curl -X 'PUT' \
+  '/api/policy/v1/policies/osdu/partition/osdu/search.rego' \
+  -H 'accept: application/json' \
+  -H 'correlation-id: 123' \
+  -H 'data-partition-id: osdu' \
+  -H 'Authorization: Bearer <TOKEN>' \
+  -H 'Content-Type: multipart/form-data' \
+  -F 'file=@search.rego'
+```
+
+</details>
+
+### GET /api/policy/v1/policies/{policy_id}
+
+Retrieves the policy with the give policy_id. This API returns the definition of the requested policy and it includes the policy definitions in the raw Rego form. It performs authorization check. The user making the call needs to be either _service.policy.user_ or _service.policy.admin_ in the provided data partition.
+
+<details>
+
+```
+curl -X 'GET' \
+  '/api/policy/v1/policies/osdu/instance/search' \
+  -H 'accept: application/json' \
+  -H 'correlation-id: 123' \
+  -H 'data-partition-id: osdu' \
+  -H 'Authorization: Bearer <TOKEN>'
+```
+
+</details>
+
+### DELETE /api/policy/v1/policies/{policy_id}
+
+Deletes the policy with the give policy_id. The API performs authorization check. The user making the call needs to be a _service.policy.admin_ in the provided data partition.
+
+<details>
+
+```
+curl -X 'DELETE' \
+  '/api/policy/v1/policies/osdu/partition/osdu/search.rego' \
+  -H 'accept: application/json' \
+  -H 'correlation-id: 123' \
+  -H 'data-partition-id: osdu' \
+  -H 'Authorization: Bearer <TOKEN>'
+```
+
+</details>
+
+## Policy APIs for using policies
+
+### POST /api/policy/v1/evaluations/query
+
+Evaluates the provided policy (referenced with _policy_id_) with the provided input. The payload for the evaluation contains a file with the _policy_id_ and _input_ for evaluation. Input contains the _operation_, _record_ or list of _records_, _groups_ that the user is a member of, _user_ attributes that can be used in a policy definition, and _legaltags_ that contain legal attributes for record(s). 
+ 
+<details>
+File example:
+```
+{
+    "input": {
+        "operation": "update",
+        "records": [
+            {
+                "id":"XXXX:test:1.4.1654807204111",
+                "kind":"XXXX:bulkupdate:test:1.1.1654807204111",
+                "legal":{
+                    "legaltags":[
+                        "YYYY"
+                    ],
+                    "otherRelevantDataCountries":["US"],
+                    "status":"compliant"
+                },
+                "acls":{
+                    "viewers":["data.default.viewers@XXXX.group"],
+                    "owners":["data.default.owners@XXXX.group"]
+                }
+            }
+        ]
+    }
+}
+```
+
+Curl:
+```
+curl -X 'POST' \
+  '/api/policy/v1/evaluations/query?policy_id=search2.rego&include_auth=true' \
+  -H 'accept: application/json' \
+  -H 'correlation-id: 123' \
+  -H 'data-partition-id: osdu' \
+  -H 'Authorization: Bearer <TOKEN>' \
+  -H 'Content-Type: multipart/form-data' \
+  -F 'file=@eval_input_allow.json;type=application/json'
+```
+
+</details>
+
+### POST /api/policy/v1/translate
+
+Translates the provided policy (referenced in query) with the provided input. The payload for the translate contains a json with the _query_, _input_ and unknowns for translation. Input contains the _operation_, and optionally can contain _groups_ that the user is a member of, xuserid to identify the user, as well as the token and data partition.
+
+Also see [translate documentation](translate.md)
+<details>
+
+```
+curl -X 'POST' \
+  '/api/policy/v1/translate' \
+  -H 'accept: application/json' \
+  -H 'correlation-id: 123' \
+  -H 'data-partition-id: osdu' \
+  -H 'Authorization: Bearer <TOKEN>' \
+  -H 'Content-Type: application/json' \
+  -d '{
+        "query": "data.osdu.partition[\"osdu\"].search.allow == true",
+        "input": {
+            "operation": "view",
+            "groups": ["AAA", "BBB"]
+        },
+        "unknowns": ["input.record"]
+}'
+```
+
+</details>
+
+## Diagnostic Policy APIs
+
+### GET /api/policy/v1/info
+
+<details>
+
+```
+curl -X 'GET' \
+  '/api/policy/v1/info' \
+  -H 'accept: application/json' \
+  -H 'correlation-id: 123'
+```
+</details>
+
+### GET /api/policy/v1/health
+
+<details>
+
+Note some providers may require authentication headers.
+```
+curl -X 'GET' \
+  'http://localhost:8080/api/policy/v1/health' \
+  -H 'accept: application/json' \
+  -H 'correlation-id: 124'
+```
+
+</details>
+
+### GET /api/policy/v1/ready
+
+<details>
+
+Note some providers may require authentication headers.
+```
+curl -X 'GET' \
+  '/api/policy/v1/ready' \
+  -H 'accept: application/json' \
+  -H 'correlation-id: 123'
+```
+
+</details>
+
+### GET /diag/policies
+
+Note this API requires `ENABLE_DEV_DIAGNOSTICS` to be enabled. This should not be enabled in production environments.
+
+<details>
+
+
+```
+curl -X 'GET' \
+  '/diag/policies' \
+  -H 'accept: application/json' \
+  -H 'correlation-id: 123' \
+  -H 'data-partition-id: osdu' \
+  -H 'Authorization: Bearer <TOKEN>'
+```
+
+</details>
+
+### GET /diag/config
+
+Note this API requires `ENABLE_DEV_DIAGNOSTICS` to be enabled. This should not be enabled in production environments.
+
+<details>
+
+
+```
+curl -X 'GET' \
+  '/diag/config' \
+  -H 'accept: application/json' \
+  -H 'correlation-id: 123' \
+  -H 'data-partition-id: osdu' \
+  -H 'Authorization: Bearer <TOKEN>'
+```
+
+</details>
+
+## Policy Service Utility APIs
+
+### POST /api/policy/v1/compile
+
+The Compile API allows you to partially evaluate Rego queries and obtain a simplified version of the policy.
+
+<details>
+
+```
+curl -X 'POST' \
+  '/api/policy/v1/compile?metrics=false&instrument=false' \
+  -H 'accept: application/json' \
+  -H 'correlation-id: 123' \
+  -H 'data-partition-id: osdu' \
+  -H 'Authorization: Bearer <TOKEN>' \
+  -H 'Content-Type: multipart/form-data' \
+  -F 'file=@compile_input1.json;type=application/json'
+```
+
+</details>
+
+#### Metrics
+
+When query parameter metrics=true, the API response will include detailed performance metrics from OPA.
+
+<details>
+
+OPA currently supports the following query performance metrics:
+
+- timer_rego_input_parse_ns: time taken (in nanoseconds) to parse the input
+- timer_rego_query_parse_ns: time taken (in nanonseconds) to parse the query.
+- timer_rego_query_compile_ns: time taken (in nanonseconds) to compile the query.
+- timer_rego_query_eval_ns: time taken (in nanonseconds) to evaluate the query.
+- timer_rego_module_parse_ns: time taken (in nanoseconds) to parse the input policy module.
+- timer_rego_module_compile_ns: time taken (in nanoseconds) to compile the loaded policy modules.
+- timer_server_handler_ns: time take (in nanoseconds) to handle the API request.
+
+</details>
+
+#### Instrumentation
+
+To enable query instrumentation, specify metrics=true and instrument=true query parameters when executing the API call. Query instrumentation can help diagnose performance problems, however, it can add significant overhead to query evaluation. We recommend leaving query instrumentation off unless you are debugging a performance problem.
+
+### GET /api/policy/v1/backup
+
+Experimental Backup API.  Allows downloading the bundle for a data partition.
+
+<details>
+
+```
+curl -X 'GET' \
+  '/api/policy/v1/backup' \
+  -H 'accept: application/json' \
+  -H 'correlation-id: 123' \
+  -H 'data-partition-id: osdu' \
+  -H 'Authorization: Bearer <TOKEN>' \
+> --output backup-bundle-osdu.tar.gz
+```
+
+</details>
+
+### GET /api/policy/v1/bootstrap
+
+Experimental Bootstrap API for creating (or reseting) bundle for a data partition.
+
+<details>
+
+```
+curl -X 'POST' \
+  '/api/policy/v1/bootstrap?force=true' \
+  -H 'accept: application/json' \
+  -H 'correlation-id: 123' \
+  -H 'data-partition-id: osdu' \
+  -H 'Authorization: Bearer <TOKEN>' \
+  -d ''
+```
+
+</details>
+
+### Permissions
+
+| **_Endpoint URL_** | **_Method_** | **_Minimum Permissions Required_** |
+| --- | --- | --- |
+| /api/policy/v1/policies | GET | service.policy.user |
+| /api/policy/v1/policies/{policy_-_id} | PUT | service.policy.admin |
+| /api/policy/v1/policies/{policy_-_id} | DELETE | service.policy.admin |
+| /api/policy/v1/policies/{policy_-_id} | GET | service.policy.user |
+| /api/policy/v1/evaluations/query | POST | service.policy.user |
 
 ---
 
+[historical information](https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/wikis/Policy-service)
+
+---
+
+## Interactive 
+
 For interactive swagger I recommend using [AdminCLI Browser command](https://osdu.pages.opengroup.org/ui/admincli/commands_reference/#adminclipy-browser)
 
-## Swagger
+### Swagger
 
 <div class="termy">
 ```console
@@ -14,7 +335,7 @@ $ admincli.py browser /api/policy/v1/docs
 ```
 </div>
 
-## Redoc
+### Redoc
 
 <div class="termy">
 ```console
@@ -22,7 +343,7 @@ $ admincli.py browser /api/policy/v1/redoc
 ```
 </div>
 
-## OpenAPI json
+### OpenAPI json
 
 <div class="termy">
 ```console

docs/docs/index.md

@@ -4,6 +4,16 @@
 
 Policy service is an OSDU service used to manage (view, create, update, delete) and evaluate dynamic policies. Dynamic policies are written using [Rego](https://www.openpolicyagent.org/docs/latest/policy-language/) language. Rego queries are assertions on data. It is a declarative language so it is convenient for writing policies.
 
+OSDU data platform has two additional service that are used together with policy service to perform data authorization:
+- [**Entitlements service**](https://community.opengroup.org/osdu/documentation/-/wikis/Releases/R2.0/Services/Entitlements/Tutorial) which provides information about user group membership,
+- [**Legal (aka compliance) service**](https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/blob/master/docs/tutorial/ComplianceService.md) that provides additional attributes that can be used to conform to legal and organizational regulations.
+
+OSDU data platform user that has required permissions (e.g., OSDU admin) can use information about:
+- **Data** or more specifically record metadata that contains _id_, _kind_, _acl_, _legal_, and _tags_;
+- **Entitlements groups** which correspond to flattened list of all groups user is a member of and can be used in record _acl_;
+- **Legal tags** which are referenced from record _legal_ block; 
+to write complex expressions defining who can discover and access what data.
+
 Last updated on {{ git_revision_date }}
 
 ---

docs/docs/releasenotes.md

@@ -10,11 +10,13 @@ None
 - Logging changes
 - Support for comments in test templates used in integration testing
 - OPA recommended version is 0.54 to address numerous bugs, security and performance issues.
+- Allow caching to be disabled via environment variable `DISABLE_OPA_CACHE`
 
 ### M18 Features
 - Translate API pre-processor (enabled by default, but can be disabled by conf.ENABLE_TRANSLATE_PREPROCESSOR). This allows Policy translate (which is used by search) to support more [complex queries](https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/82) and [deny policies](https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/76)
 - Translate API change - groups are now optional in Input of [TranslateItem](https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/blob/master/app/models/translate.py). See updated [openapi.yaml](https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/blob/master/docs/openapi.yaml). This should make this API easier to use and increase performance.
-- Allow caching to be disabled via environment variable `DISABLE_OPA_CACHE`
+- Backup API [AWS](https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/36) and [Azure](https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/35)
+- Bootstrap API to for [adding new partitions](https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/94)
 
 ### M18 Issues
 

docs/openapi.yaml

 openapi: 3.0.2
 info:
-  title: Policy Service
+  title: OSDU Policy Service
   description: >2
 
-        OSDU Policy Service v0.1.4 API 
+        OSDU Policy Service v0.1.4 API for Milestone M18
 
     Policy service is used for management and evaluation of dynamic policies in
     OSDU.
@@ -27,15 +27,23 @@ info:
     _service.policy.admin_ group.
 
     * User can be added to these groups by using entitlements service.
+
+
+    ### Useful information
+
+    *
+    [Documentation](https://osdu.pages.opengroup.org/platform/security-and-compliance/policy/)
+
+    *
+    [Source](https://community.opengroup.org/osdu/platform/security-and-compliance/policy)
   contact:
     name: OSDU Policy Service
-    url: >-
-      https://community.opengroup.org/osdu/platform/security-and-compliance/policy
+    url: https://osdu.pages.opengroup.org/platform/security-and-compliance/policy/
   license:
     name: Apache 2.0
     url: >-
       https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/raw/trusted-FastAPI/LICENSE
-  version: 0.1.4
+  version: v0.1.4 M18
 paths:
   /:
     get:
@@ -779,7 +787,7 @@ paths:
   /api/policy/v1/compile:
     post:
       tags:
-        - diagnostic
+        - utility
       summary: Compile Partially Evaluate A Query
       description: >-
         # Compile - Partially evaluate a query.
@@ -991,9 +999,17 @@ paths:
   /api/policy/v1/backup:
     get:
       tags:
-        - diagnostic
+        - utility
       summary: Backup
-      description: Experimental Backup API.
+      description: >-
+        Experimental Backup API.
+
+
+        Allows downloading the bundle for a data partition.
+
+
+        Bundle filename will be in the form bundle-`data
+        partition`-`date`.tar.gz
       operationId: backup_api_policy_v1_backup_get
       parameters:
         - required: false
@@ -1029,6 +1045,137 @@ paths:
       responses:
         '200':
           description: Successful Response
+          content:
+            application/json:
+              schema: {}
+        '422':
+          description: Validation Error
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/HTTPValidationError'
+      security:
+        - HTTPBearer: []
+  /api/policy/v1/bootstrap:
+    post:
+      tags:
+        - utility
+      summary: Bootstrap
+      description: |-
+        Experimental bootstrap API for creating and updating bundle to default.
+        This should be used when adding a partition to OSDU.
+
+        Without force:
+
+            * This method is only allowed if the partition doesn't already have a bundle.
+            * If the bundle already exists it will return 405 METHOD_NOT_ALLOWED.
+            * Policy Service can be configured to ignore force.
+
+        May return:
+
+            * HTTP_202_ACCEPTED - updated
+            * HTTP_201_CREATED - created
+            * HTTP_405_METHOD_NOT_ALLOWED - not allowed
+            * HTTP_424_FAILED_DEPENDENCY - bundle server caused failure
+      operationId: bootstrap_api_policy_v1_bootstrap_post
+      parameters:
+        - required: false
+          schema:
+            title: Force
+            type: boolean
+            default: false
+          name: force
+          in: query
+        - required: false
+          schema:
+            title: Correlation-Id
+            type: string
+          name: correlation-id
+          in: header
+        - required: false
+          schema:
+            title: User-Agent
+            type: string
+          name: user-agent
+          in: header
+        - description: identifier of the data partition to query
+          required: false
+          schema:
+            title: Data-Partition-Id
+            minLength: 1
+            type: string
+            description: identifier of the data partition to query
+          name: data-partition-id
+          in: header
+        - description: identifier the user in the query
+          required: false
+          schema:
+            title: X-User-Id
+            minLength: 1
+            type: string
+            description: identifier the user in the query
+          name: x-user-id
+          in: header
+      responses:
+        '200':
+          description: Successful Response
+          content:
+            application/json:
+              schema: {}
+        '422':
+          description: Validation Error
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/HTTPValidationError'
+      security:
+        - HTTPBearer: []
+  /api/policy/v1/login:
+    get:
+      tags:
+        - login
+      summary: Login
+      description: |-
+        Login method for future use with Admin UI.
+        Requires authorized admin user.
+      operationId: login_api_policy_v1_login_get
+      parameters:
+        - required: false
+          schema:
+            title: Correlation-Id
+            type: string
+          name: correlation-id
+          in: header
+        - required: false
+          schema:
+            title: User-Agent
+            type: string
+          name: user-agent
+          in: header
+        - description: identifier of the data partition to query
+          required: false
+          schema:
+            title: Data-Partition-Id
+            minLength: 1
+            type: string
+            description: identifier of the data partition to query
+          name: data-partition-id
+          in: header
+        - description: identifier the user in the query
+          required: false
+          schema:
+            title: X-User-Id
+            minLength: 1
+            type: string
+            description: identifier the user in the query
+          name: x-user-id
+          in: header
+      responses:
+        '200':
+          description: Successful Response
+          content:
+            application/json:
+              schema: {}
         '422':
           description: Validation Error
           content:
@@ -1481,8 +1628,14 @@ tags:
     description: Evaluate APIs
   - name: translate
     description: Translation APIs
+    externalDocs:
+      description: Translate API documentation
+      url: >-
+        https://osdu.pages.opengroup.org/platform/security-and-compliance/policy/translate/
   - name: diagnostic
     description: Version information and other diagnostic APIs
+  - name: utility
+    description: Service APIs
   - name: login
     description: Login for Admin UI
   - name: home