Policy issueshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues2023-03-06T23:22:01Zhttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/83[Google] Policy Service has high latency2023-03-06T23:22:01ZYan Sushchynski (EPAM)[Google] Policy Service has high latencyHello,
We'd appreciate your support.
We faced high latency of Policy service on Google, and the latency is about 8 minute.
![image](/uploads/80ba384f6af0a6d5a4e8894f821a2ab2/image.png)
In addition, Policy service usually works for a...Hello,
We'd appreciate your support.
We faced high latency of Policy service on Google, and the latency is about 8 minute.
![image](/uploads/80ba384f6af0a6d5a4e8894f821a2ab2/image.png)
In addition, Policy service usually works for a while and then it starts throwing `502`-error, and then it needs to be restarted.
Also, we can see that Policy service is constantly requested with `POST /api/policy/v1/translate` request.M15 - Release 0.18Shane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/126Update Policy Service API version2024-01-25T16:17:12ZShane HutchinsUpdate Policy Service API versionMake Policy Service API version to be 1.0.0 and not include any other identifiers (including v, milestone, build information, etc).
The Build information is still available in description and info API.
Requested by @chadMake Policy Service API version to be 1.0.0 and not include any other identifiers (including v, milestone, build information, etc).
The Build information is still available in description and info API.
Requested by @chadM23 - Release 0.26Shane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/125Upgrade SHA1 to SHA2 or SHA2562024-02-13T14:11:22ZShane HutchinsUpgrade SHA1 to SHA2 or SHA256Policy Service currently uses SHA1 for logging purposes with changes to policies. This SHA1 is returned in json response when changes are made as well.
While it's not used for security it would be nice to upgrade from SHA1.
Created bas...Policy Service currently uses SHA1 for logging purposes with changes to policies. This SHA1 is returned in json response when changes are made as well.
While it's not used for security it would be nice to upgrade from SHA1.
Created based upon https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/124Shane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/124bandit scan issue: Use of weak SHA1 hash for security.2024-02-13T01:10:20ZSolomon Ayalewbandit scan issue: Use of weak SHA1 hash for security.bandit scan is showing a potential issue with Severity: High, Confidence: High
check the scan log for detail.
Run started:2023-12-18 22:51:23.816146
```
Test results:
>> Issue: [B324:hashlib] Use of weak SHA1 hash for security. Consid...bandit scan is showing a potential issue with Severity: High, Confidence: High
check the scan log for detail.
Run started:2023-12-18 22:51:23.816146
```
Test results:
>> Issue: [B324:hashlib] Use of weak SHA1 hash for security. Consider usedforsecurity=False
Severity: High Confidence: High
CWE: CWE-327 (https://cwe.mitre.org/data/definitions/327.html)
More Info: https://bandit.readthedocs.io/en/1.7.6/plugins/b324_hashlib.html
Location: /Users/solxget/OSDU-clean/os-policy-service/app/api/policy_read_api.py:317:23
316 data = opa_response.json["result"]["raw"]
317 sha1 = hashlib.sha1(data.encode()).hexdigest()
318 response.headers["X-SHA-1"] = sha1
--------------------------------------------------
>> Issue: [B324:hashlib] Use of weak SHA1 hash for security. Consider usedforsecurity=False
Severity: High Confidence: High
CWE: CWE-327 (https://cwe.mitre.org/data/definitions/327.html)
More Info: https://bandit.readthedocs.io/en/1.7.6/plugins/b324_hashlib.html
Location: /Users/solxget/OSDU-clean/os-policy-service/app/api/policy_update_api.py:325:11
324
325 sha1 = hashlib.sha1(contents.decode("utf-8").encode()).hexdigest()
326 response.headers["X-SHA-1"] = sha1
--------------------------------------------------
>> Issue: [B324:hashlib] Use of weak SHA1 hash for security. Consider usedforsecurity=False
Severity: High Confidence: High
CWE: CWE-327 (https://cwe.mitre.org/data/definitions/327.html)
More Info: https://bandit.readthedocs.io/en/1.7.6/plugins/b324_hashlib.html
Location: /Users/solxget/OSDU-clean/os-policy-service/app/api/validate_api.py:96:15
95 ):
96 sha1 = hashlib.sha1(data.encode()).hexdigest()
97 response.headers["X-SHA-1"] = sha1
--------------------------------------------------
>> Issue: [B324:hashlib] Use of weak SHA1 hash for security. Consider usedforsecurity=False
Severity: High Confidence: High
CWE: CWE-327 (https://cwe.mitre.org/data/definitions/327.html)
More Info: https://bandit.readthedocs.io/en/1.7.6/plugins/b324_hashlib.html
Location: /Users/solxget/OSDU-clean/os-policy-service/app/bundles/bundle.py:156:44
155 contents = f.read()
156 existing_sha1 = hashlib.sha1(contents).hexdigest()
157 updated_existing = True
--------------------------------------------------
>> Issue: [B324:hashlib] Use of weak SHA1 hash for security. Consider usedforsecurity=False
Severity: High Confidence: High
CWE: CWE-327 (https://cwe.mitre.org/data/definitions/327.html)
More Info: https://bandit.readthedocs.io/en/1.7.6/plugins/b324_hashlib.html
Location: /Users/solxget/OSDU-clean/os-policy-service/app/bundles/bundle.py:161:35
160 if updated_existing:
161 updated_sha1 = hashlib.sha1(policy).hexdigest()
162 if existing_sha1 == updated_sha1:
--------------------------------------------------
Code scanned:
Total lines of code: 7294
Total lines skipped (#nosec): 0
Total potential issues skipped due to specifically being disabled (e.g., #nosec BXXX): 0
Run metrics:
Total issues (by severity):
Undefined: 0
Low: 138
Medium: 34
High: 6
Total issues (by confidence):
Undefined: 0
Low: 33
Medium: 3
High: 142
Files skipped (0):
```M22 - Release 0.25David Diederichd.diederich@opengroup.orgChad LeongShane HutchinsDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/123Documentation update request Arch Diagram2023-11-30T19:04:10ZShane HutchinsDocumentation update request Arch DiagramThe Diagram https://osdu.pages.opengroup.org/platform/security-and-compliance/policy/arch/ is slightly outdated.
This should be updated to reflect M21/M22
Requested by @MonicaJohnsThe Diagram https://osdu.pages.opengroup.org/platform/security-and-compliance/policy/arch/ is slightly outdated.
This should be updated to reflect M21/M22
Requested by @MonicaJohnsM22 - Release 0.25Shane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/122Policy validation issue for non-ascii characters2023-11-16T17:32:01ZDadong ZhouPolicy validation issue for non-ascii charactersIn M21, the new policy validation has issue for non-ascii characters. Here is a sample policy tested in M21 GC:
```
package osdu.partition["m19"].organisation_code_2
organisation_code := {
"AGÊNCIA NACIONAL DO PETRÓLEO": {
"Name":...In M21, the new policy validation has issue for non-ascii characters. Here is a sample policy tested in M21 GC:
```
package osdu.partition["m19"].organisation_code_2
organisation_code := {
"AGÊNCIA NACIONAL DO PETRÓLEO": {
"Name": "ANP",
"Code": "G0013"
}
}
```
Failed to load the policy with the following error:
```
{
"detail":"Unable to validate policy! Error: {\n \"code\": \"invalid_parameter\",\n \"message\": \"error(s) occurred while compiling module(s)\",\n \"errors\": [\n {\n \"code\": \"rego_parse_error\",\n \"message\": \"unexpected assign token: expected rule value term (e.g., organisation_code := \<VALUE\> { ... })\",\n \"location\": {\n \"file\": \"tmp/m19/organisation_code_2.rego\",\n \"row\": 3,\n \"col\": 19\n },\n \"details\": {\n \"line\": \"organisation_code := {\",\n \"idx\": 18\n }\n }\n ]\n}\n 400."
}
```
This policy can be loaded before the policy validation is added in M21.
@hutchins @KellyZhouM22 - Release 0.25Shane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/121Policy Service APIs should use introduce a new group2023-11-16T17:31:45ZShane HutchinsPolicy Service APIs should use introduce a new group- Allow operations related APIs to use a new group 'service.policy.ops' (but make it configurable).
- For backwards compatibility make it default to 'service.policy.admin'.
For the following APIs:
- GET, PUT, DELETE /api/policy/v1/tenan...- Allow operations related APIs to use a new group 'service.policy.ops' (but make it configurable).
- For backwards compatibility make it default to 'service.policy.admin'.
For the following APIs:
- GET, PUT, DELETE /api/policy/v1/tenant
Maybe also:
- GET /api/policy/v1/backup
- POST /api/policy/v1/bootstrapM22 - Release 0.25Shane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/120Pipeline test automation /health API forbidden 403 on IBM2023-10-30T13:29:51ZShane HutchinsPipeline test automation /health API forbidden 403 on IBM- update integration tests to allow 403 on /health- update integration tests to allow 403 on /healthM22 - Release 0.25Shane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/119Policy /tenant API bug2023-10-30T13:32:56ZShane HutchinsPolicy /tenant API bug- service parameter not limited to supported values
- polling_min_delay_seconds not checked
- polling_max_delay_seconds not checked- service parameter not limited to supported values
- polling_min_delay_seconds not checked
- polling_max_delay_seconds not checkedM22 - Release 0.25Shane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/118Policy Service not all return status codes that are specified in OpenAPI.json2023-11-16T17:32:36ZShane HutchinsPolicy Service not all return status codes that are specified in OpenAPI.jsonFound during preship testing of M21
Policy Service openapi json should be updated to include all possible status codes.Found during preship testing of M21
Policy Service openapi json should be updated to include all possible status codes.M22 - Release 0.25Shane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/117The policy service allows to break a search in the partition2023-10-31T22:10:19ZOleksandr Kosse (EPAM)The policy service allows to break a search in the partitionWe've figured out that uploading a policy rule that blocks all searches in the tenant is possible. Policy service for such cases doesn't provide any tools for verification of the new rule, troubleshooting the existing rules, or rollback ...We've figured out that uploading a policy rule that blocks all searches in the tenant is possible. Policy service for such cases doesn't provide any tools for verification of the new rule, troubleshooting the existing rules, or rollback of the applied rules. That leads to potential issues with any new production environments.
Please consider the following solutions:
- Add to the policy service a checker for new rules
- Add a history for all applied rules with rollback options
- disable creating/updating policy rules via API and delegate it on the infrastructure level (as legal service uses)
I really appreciate any other options that allow us to manage policy service in the production environmentM22 - Release 0.25Shane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/115Policy service isn't work propely after integration tests2023-10-30T14:40:39ZYauheni Rykhter (EPAM)Policy service isn't work propely after integration testsHello!
During our pipeline, we have the step when we deploy Policy service and initialization bundles (see the example):
![image](/uploads/5f5b05870bbea7937bdc09d716c1b5a5/image.png).
After integration tests, you can see that the size o...Hello!
During our pipeline, we have the step when we deploy Policy service and initialization bundles (see the example):
![image](/uploads/5f5b05870bbea7937bdc09d716c1b5a5/image.png).
After integration tests, you can see that the size of bundle was changed (see the example):
![image](/uploads/087374c11b7716116d8445fcfd9063b7/image.png).
Can you update tests that leave bundles as they were initialized?
Thanks in advance!
JFYI, @Oleksandr_Kosse, @Yauhen_ShaliouShane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/114Job Failed #22795812023-10-13T17:18:08ZYan Sushchynski (EPAM)Job Failed #2279581May I ask for your help with the job [#2279581](https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/jobs/2279581) (failed for 683a03119c3ce0695f87a536fbfb0cd0bd042a88).
The logs don't show any errors since Po...May I ask for your help with the job [#2279581](https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/jobs/2279581) (failed for 683a03119c3ce0695f87a536fbfb0cd0bd042a88).
The logs don't show any errors since Policy service returns unexpected OK statuses. Possibly this issue can affect Indexer serviceM21 - Release 0.24Shane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/113Policy Service should have a separate audit log2024-02-13T14:11:41ZShane HutchinsPolicy Service should have a separate audit log@MonicaJohns requested that Policy Service should have it's own audit log (in addition to the information gathered in pod logs).
Thinking I could save a file in bundle server (S3, blob storage) with each change or something like that.@MonicaJohns requested that Policy Service should have it's own audit log (in addition to the information gathered in pod logs).
Thinking I could save a file in bundle server (S3, blob storage) with each change or something like that.M23 - Release 0.26Shane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/112Policy Service should handle entitlement service issues more gracefully2023-10-09T18:02:18ZShane HutchinsPolicy Service should handle entitlement service issues more gracefullyIf entitlement is not available or returns 5xx policy service may not handle the error gracefully.If entitlement is not available or returns 5xx policy service may not handle the error gracefully.M21 - Release 0.24Shane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/111Policy Service bundle changes even if there isn't really a change to policies2023-10-04T03:50:45ZShane HutchinsPolicy Service bundle changes even if there isn't really a change to policiesPolicy Service updates the bundle for any valid policy put and delete today.
It even updates the bundle even if the policy "changing" has no changes.
I think policy service should only update the bundle if the actual policy changes.
If ...Policy Service updates the bundle for any valid policy put and delete today.
It even updates the bundle even if the policy "changing" has no changes.
I think policy service should only update the bundle if the actual policy changes.
If it changes return 202, if it's the same return 200.
This is not considered a bug, but a feature enhancement.
@dadong.zhouM21 - Release 0.24Shane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/110Request to have a M18 patch on AWS for memory leak fix2023-09-20T21:40:20ZDadong ZhouRequest to have a M18 patch on AWS for memory leak fixShell would like to request to have a M18 patch for Policy service memory leak fix. The memory leak issue with test results is documented in: https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/93.
As s...Shell would like to request to have a M18 patch for Policy service memory leak fix. The memory leak issue with test results is documented in: https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/93.
As shown in the test results, the memory leak is caused by using the logging package "coloredlogs" of version 15.0.1. It is tested that using an old version 14.2 of the package fixes the memory leak. The logging package is completely removed in M20.
We would like to request to have the M18 patch available on AWS with the only change of the logging package version number.
Thanks.
cc @KellyZhou @hutchins @MonicaJohnsShane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/109Mask OPA Configmap credentials2023-08-29T18:32:34ZShane HutchinsMask OPA Configmap credentialsRemove reporting on of credentials section of config map from /config API for security reasons.
@KellyZhouRemove reporting on of credentials section of config map from /config API for security reasons.
@KellyZhouM20 - Release 0.23Shane HutchinsShane Hutchinshttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/103Add option in Translate Api to construct elastic search subquery in search re...2023-08-22T18:08:51ZDadong ZhouAdd option in Translate Api to construct elastic search subquery in search rego policy@hmarkovic @hutchins @srabanaguha
We have added the preprocessor in Translate Api in M18 to handle the http calls and the allow/deny rules in search policy. The current translate logic is still limited to simple Rego syntax. To close t...@hmarkovic @hutchins @srabanaguha
We have added the preprocessor in Translate Api in M18 to handle the http calls and the allow/deny rules in search policy. The current translate logic is still limited to simple Rego syntax. To close the gap, we can add an option in the preprocessor to allow the search policy to construct its own elastic search subquery in Rego files. The current Translate api needs be updated to check the preprocessor results for the new proposed optional field "es_subquery":
```
preprocess_config := {
"input_from_preprocessor": {
# any results from the preprocessor evaluation that will be used by the allow/deny rules
},
"has_allow_rule": true/false,
"has_deny_rule": true/false,
"es_subquery": {"query": {...}} # new proposed optional field
}
```
If the "es_subquery" field exists, the Translate api will skip the compile/translate logic and simply return this field back to the search service.
And here is a search policy example:
```
package osdu.partition["osdu"].search
import data.osdu.partition["osdu"].search_preprocessor
preprocess_config := {
"es_subquery": search_preprocessor.es_subquery
}
```
```
package osdu.partition["osdu"].search_preprocessor
# Search policy example to allow search all data records
es_subquery := {
"query": {
"match_all": {}
}
}
```
```
package osdu.partition["osdu"].search_preprocessor
# Search policy example to deny search any data records
es_subquery := {
"query": {
"match_none": {}
}
}
```
```
package osdu.partition["osdu"].search_preprocessor
# Search policy example to search data records for a particular data group
es_subquery := {
"query": {
"bool": {
"should": [
{
"bool": {
"filter": [
{
"term": {
"acl.viewers": "data.site.administrators"
}
}
]
}
},
{
"bool": {
"filter": [
{
"term": {
"acl.owners": "data.site.administrators"
}
}
]
}
}
]
}
}
}
```M20 - Release 0.23Srabana GuhaSrabana Guhahttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/101Minor issue - translate api for deny rule only2023-08-29T18:13:22ZDadong ZhouMinor issue - translate api for deny rule onlyI am working on Pre-Shipping postman collection for the translate api M18 enhancement. I am testing on Azure. We missed a special scenario: when having no Allow rule and only having Deny rule. The return query is {"bool":{"must_not":[ .....I am working on Pre-Shipping postman collection for the translate api M18 enhancement. I am testing on Azure. We missed a special scenario: when having no Allow rule and only having Deny rule. The return query is {"bool":{"must_not":[ ...]}}. It needs be {"query": {"bool":{"must_not":[...]}}}. We missed the start "query" field. This is a very special scenario and we don't need the fix in M18 if build/deploy to M18 is complex.M20 - Release 0.23Srabana GuhaSrabana Guha