AWS Parameterizing OPA Cache Settings as Helm Values and Right-Sizing Default OPA Cache Size

devops/aws/opa/templates/configmap.yaml

@@ -16,12 +16,17 @@
 {{- $policyBucket  := .Values.policyBucket -}}
 {{- $minDelay  := .Values.minPollingDelaySeconds -}}
 {{- $maxDelay  := .Values.maxPollingDelaySeconds -}}
+{{- $cacheSizeBytes  := .Values.cacheSizeBytes -}}
+{{- $cacheEvictionPeriodSeconds  := .Values.cacheEvictionPeriodSeconds -}}
 
 metadata:
   name: opa-config
 data:
   config: |
-    caching.inter_query_builtin_cache.max_size_bytes: 943718400
+    caching:
+      inter_query_builtin_cache:
+        max_size_bytes: {{ default 400000000 .Values.cacheSizeBytes | int }}
+        stale_entry_eviction_period_seconds: {{ default 60 .Values.cacheEvictionPeriodSeconds | int }}
     services: 
       s3:
         url: {{ .Values.policyBucket }}

docs/docs/sizing.md

@@ -31,5 +31,6 @@ minReplicas should always be 3 or more in production environments or preship env
 
 - For OPA minimum sizing cpu requests 500M, memory 256Mi and 2 pods.
 - Recommended is 1000M CPU and 1000Mi RAM.
+- Recommend to use less than 400000000 max_size_bytes setting for the OPA cache with 1000Mi memory.
 - Horizontal Pod Autoscaling (HPA) is recommended and minReplicas should always be 2 or more in production environments and pre-ship environments.  It has been reported that 1000M CPU, 1000Mi Memory requests and a minimum of 3 pods works well under load.
 - The number of OPA pods should be *at least* half the number of policy service pods. i.e. if you have 6 policy pods, you should have 3 OPA pods.
\ No newline at end of file