Address Performance Gap
The last run of storage and search performance with the policy service enabled shows:
- Storage performance is 80-100% lower with Policy service enabled.
- Search performance is 20-40% lower with Policy service enabled.
- We starting to get errors with 50 concurrent request threads (500 Internal Service Error).
Next Steps
-
Clarify scenarios, data and environment -
Test and Document Baseline Performance -
Identify bottlenecks and root cause (use Policy Profiling?) Current assessment is that Every call is going out to entitlements and legal that's the source of the issue -
Update design to: - Leverage OPA - loading bundles to server
- OPA scaling
- Cache entitlements and legal tags in OPA
- Options store as bundle
- Use OPA request / caching
-
Implement Changes -
Run tests and compare to baseline
Links
Persisting policies and autoscaling for CSPs
Challenges/Needs
- may Need Java Devs to help with integration with the other Java services (the policy service is currently written in Python)
- Deployment is also a problem - bundling is a problem. Not able to do continuous deployment