OPA http requests calling own policy service apis are blocked
While testing a policy, it is found that the http request in Rego policy is failing when calling the Storage api to retrieve a data record.
It is further investigated with a test rego policy using different urls. It seems the http requests calling own policy service apis are blocked.
Here are the tests performed (For test cases 1 - 4, the test policy is deployed and evaluated in Shell OSDU Sandbox; For cases 5 & 6, the test policy is deployed and evaluated in my local machine Policy instance):
Policy rego file:
package osdu.partition["osdu"].test
import input
headers = {
"Content-Type": "application/json",
"data-partition-id":input.datapartitionid,
"Authorization": sprintf("Bearer %v", [input.token]),
"Accept": "application/json"
}
url := input.url
response := http.send({
"method": "GET",
"url": url,
"headers": headers,
"force_cache": true,
"force_cache_duration_seconds": 1,
"raise_error": false
})Policy evaluation case 1:
Input - call Storage Info api:
{
"input": {
"url": "https://sandbox.osdu.shell.com/api/storage/v2/info"
}
}Evaluation output with expected results (status code 200) - http call is working:
"response": {
"body": {
"artifactId": "storage-aws",
"branch": "refs/heads/release/r3-m15",
"buildTime": "2023-01-05T21:49:57.391Z",
"commitId": "343b1cd6109bb2c329dfa2d6c01efca241bb6688",
"commitMessage": "Merge branch 'cherry-pick-for-539' into 'release/0.18'",
"connectedOuterServices": [],
"groupId": "org.opengroup.osdu",
"version": "0.18.0-SNAPSHOT"
},
"headers": {
"access-control-allow-credentials": [
"true"
],
"access-control-allow-headers": [
"access-control-allow-origin, origin, content-type, accept, authorization, data-partition-id, correlation-id, appkey"
],
"access-control-allow-methods": [
"GET, POST, PUT, DELETE, OPTIONS, HEAD, PATCH"
],
"access-control-allow-origin": [
"*"
],
"access-control-max-age": [
"3600"
],
"cache-control": [
"no-cache, no-store, must-revalidate"
],
"content-security-policy": [
"default-src 'self'"
],
"content-type": [
"application/json"
],
"correlation-id": [
"da2ac137-8bfe-48ba-9856-dd659e1639be"
],
"date": [
"Thu, 01 Jun 2023 18:29:01 GMT"
],
"expires": [
"0"
],
"strict-transport-security": [
"max-age=31536000; includeSubDomains"
],
"x-content-type-options": [
"nosniff"
],
"x-envoy-upstream-service-time": [
"3"
],
"x-frame-options": [
"DENY"
],
"x-xss-protection": [
"1; mode=block"
]
},
"raw_body": "{\"groupId\":\"org.opengroup.osdu\",\"artifactId\":\"storage-aws\",\"version\":\"0.18.0-SNAPSHOT\",\"buildTime\":\"2023-01-05T21:49:57.391Z\",\"branch\":\"refs/heads/release/r3-m15\",\"commitId\":\"343b1cd6109bb2c329dfa2d6c01efca241bb6688\",\"commitMessage\":\"Merge branch 'cherry-pick-for-539' into 'release/0.18'\",\"connectedOuterServices\":[]}",
"status": "200 OK",
"status_code": 200
}Policy evaluation case 2:
Input - call Storage Get api with a valid data record id:
{
"input": {
"url": "https://sandbox.osdu.shell.com/api/storage/v2/records/osdu:dataset--File.Generic:PolicyTest:LT_1_OWNER"
}
}Evaluation output with unexpected results (status code 0) - http call is failing:
"response": {
"error": {
"code": "eval_http_send_network_error",
"message": "Get \"https://sandbox.osdu.shell.com/api/storage/v2/records/osdu:dataset--File.Generic:PolicyTest:LT_1_OWNER\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
},
"status_code": 0
}Policy evaluation case 3:
Input - call Storage Get api with an invalid data record id:
{
"input": {
"url": "https://sandbox.osdu.shell.com/api/storage/v2/records/osdu:dataset--File.Generic:PolicyTest:InvalidID"
}
}Evaluation output with expected results (status code 404) - http call is working:
"response": {
"body": {
"code": 404,
"message": "The record 'osdu:dataset--File.Generic:PolicyTest:InvalidID' was not found",
"reason": "Record not found"
},
"headers": {
"access-control-allow-credentials": [
"true"
],
"access-control-allow-headers": [
"access-control-allow-origin, origin, content-type, accept, authorization, data-partition-id, correlation-id, appkey"
],
"access-control-allow-methods": [
"GET, POST, PUT, DELETE, OPTIONS, HEAD, PATCH"
],
"access-control-allow-origin": [
"*"
],
"access-control-max-age": [
"3600"
],
"cache-control": [
"no-cache, no-store, must-revalidate"
],
"content-disposition": [
"inline;filename=f.txt"
],
"content-security-policy": [
"default-src 'self'"
],
"content-type": [
"application/json"
],
"correlation-id": [
"22839a83-3d4c-4a49-8ad1-f135e26a4080"
],
"date": [
"Thu, 01 Jun 2023 18:32:18 GMT"
],
"expires": [
"0"
],
"strict-transport-security": [
"max-age=31536000; includeSubDomains"
],
"x-content-type-options": [
"nosniff"
],
"x-envoy-upstream-service-time": [
"18"
],
"x-frame-options": [
"DENY"
],
"x-xss-protection": [
"1; mode=block"
]
},
"raw_body": "{\"code\":404,\"reason\":\"Record not found\",\"message\":\"The record 'osdu:dataset--File.Generic:PolicyTest:InvalidID' was not found\"}",
"status": "404 Not Found",
"status_code": 404
}Policy evaluation case 4:
Input - Call Policy Health api:
{
"input": {
"url": "https://sandbox.osdu.shell.com/api/policy/v1/health"
}
}Evaluation output with unexpected results (status code 0) - http call is failing:
"response": {
"error": {
"code": "eval_http_send_network_error",
"message": "Get \"https://sandbox.osdu.shell.com/api/policy/v1/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
},
"status_code": 0
}Policy evaluation case 5 - policy rego file is deployed and evaluated in my local machine Policy instance:
Input - same input as in case 2 - call Storage Get api with a valid data record id:
{
"input": {
"url": "https://sandbox.osdu.shell.com/api/storage/v2/records/osdu:dataset--File.Generic:PolicyTest:LT_1_OWNER"
}
}Evaluation output with expected results (status code 200) - http call is working:
"response": {
"body": {
"acl": {
"owners": [
"data.policytest.owners@osdu.shell.com"
],
"viewers": [
"data.policytest.no.viewers@osdu.shell.com"
]
},
"createTime": "2023-05-31T22:09:47.734Z",
"createUser": "osduusdevdpinformatica@shell.com",
"data": {
"DatasetProperties": {
"FileSourceInfo": {
"FileSource": "s3://osdudptfue1-shared-813258989325-us-east-1-file/osdu/uxHRxl9M8JdUf12AKuTkAZQai3LTZw6W/test.txt"
}
},
"ResourceSecurityClassification": "osdu:reference-data--ResourceSecurityClassification:RESTRICTED:"
},
"id": "osdu:dataset--File.Generic:PolicyTest:LT_1_OWNER",
"kind": "osdu:wks:dataset--File.Generic:1.0.0",
"legal": {
"legaltags": [
"osdu-Case-A1-allow-Affiliate"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"meta": [],
"modifyTime": "2023-06-01T14:37:46.994Z",
"modifyUser": "osduusdevdpinformatica@shell.com",
"version": 1685570987718344
},
"headers": {
"access-control-allow-credentials": [
"true"
],
"access-control-allow-headers": [
"access-control-allow-origin, origin, content-type, accept, authorization, data-partition-id, correlation-id, appkey"
],
"access-control-allow-methods": [
"GET, POST, PUT, DELETE, OPTIONS, HEAD, PATCH"
],
"access-control-allow-origin": [
"*"
],
"access-control-max-age": [
"3600"
],
"cache-control": [
"no-cache, no-store, must-revalidate"
],
"content-disposition": [
"inline;filename=f.txt"
],
"content-length": [
"806"
],
"content-security-policy": [
"default-src 'self'"
],
"content-type": [
"application/json"
],
"correlation-id": [
"872bccf2-1ce8-4b16-b5e9-1d3cdeba7c4f"
],
"date": [
"Thu, 01 Jun 2023 19:31:07 GMT"
],
"expires": [
"0"
],
"strict-transport-security": [
"max-age=31536000; includeSubDomains"
],
"x-content-type-options": [
"nosniff"
],
"x-envoy-upstream-service-time": [
"827"
],
"x-frame-options": [
"DENY"
],
"x-xss-protection": [
"1; mode=block"
]
},
"raw_body": "{\"data\":{\"DatasetProperties\":{\"FileSourceInfo\":{\"FileSource\":\"s3://osdudptfue1-shared-813258989325-us-east-1-file/osdu/uxHRxl9M8JdUf12AKuTkAZQai3LTZw6W/test.txt\"}},\"ResourceSecurityClassification\":\"osdu:reference-data--ResourceSecurityClassification:RESTRICTED:\"},\"meta\":[],\"id\":\"osdu:dataset--File.Generic:PolicyTest:LT_1_OWNER\",\"version\":1685570987718344,\"kind\":\"osdu:wks:dataset--File.Generic:1.0.0\",\"acl\":{\"viewers\":[\"data.policytest.no.viewers@osdu.shell.com\"],\"owners\":[\"data.policytest.owners@osdu.shell.com\"]},\"legal\":{\"legaltags\":[\"osdu-Case-A1-allow-Affiliate\"],\"otherRelevantDataCountries\":[\"US\"],\"status\":\"compliant\"},\"createUser\":\"osduusdevdpinformatica@shell.com\",\"createTime\":\"2023-05-31T22:09:47.734Z\",\"modifyUser\":\"osduusdevdpinformatica@shell.com\",\"modifyTime\":\"2023-06-01T14:37:46.994Z\"}",
"status": "200 OK",
"status_code": 200
}Policy evaluation case 6 - policy rego file is deployed and evaluated in my local machine Policy instance:
Input - same input as in case 4 - Call Policy Health api:
{
"input": {
"url": "https://sandbox.osdu.shell.com/api/policy/v1/health"
}
}Evaluation output with expected results (status code 200) - http call is working:
"response": {
"body": {
"message": "Healthy"
},
"headers": {
"content-length": [
"21"
],
"content-type": [
"application/json"
],
"date": [
"Thu, 01 Jun 2023 19:30:02 GMT"
],
"x-envoy-upstream-service-time": [
"8"
]
},
"raw_body": "{\"message\":\"Healthy\"}",
"status": "200 OK",
"status_code": 200
}In cases 1 & 3, the http calls are not calling the Policy apis and the http calls are working.
In cases 2 & 4, the http calls are calling own Policy apis directly or indirectly and the http calls are blocked and failing.
In cases 5 & 6, same inputs as in cases 2 & 4. The http calls are calling the Policy apis on different OSDU instances and the http calls are working.