Implement custom sampling for AppInsights logs to control logging costs
Problem: AppInsights log volume can get very expensive very quickly for a production environment. We'd like an optional ability to sample non-essential logs.
Use case: We'd like to sample a percentage of noisy trace and dependency logs (info
type traces and successful dependencies) while keeping all request and error logs, as well as important trace and dependency logs (i.e. audit logs (trace) and failed dependencies).
Tried but didn't help: AppInsights SDK provides log sampling for Java apps via Sampling Overrides. However, these work in a all or nothing
mode, meaning that we cannot selectively sample traces and dependencies while keeping request logs if the said traces and dependencies are logged under the same request span/context. To sample traces and dependencies, we'd have to sample requests as well, which is not what we want.
Proposed solution:
- An optional config per service that allows certain log types to be randomly sampled. This should be completely optional, meaning no breaking change and logging works exactly the same if a service doesn't explicitly opt-in for sampling.
- A service, if decides to opt-in, will provide a sampling percentage for different log types. For the initial implementation, we shall start with info and dependency sampling only. We can later extend the same principal to other tables.
- A service only needs to modify its
application.properties
file to opt-in for sampling. - Even if sampling is enabled for a service for traces,
audit
logs (which are logged in thetrace
table) should not be sampled. - Even if sampling is enabled for a service for dependencies, failed dependencies should not be sampled.
Action: MR !16 (merged)