Skip to content

Revert max_pod back to 30

Vasyl Leskiv [SLB] requested to merge vl/revert-back-max-pods into master

All Submissions:

  • [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
  • [NO] I have updated the documentation accordingly.
  • [YES] My code follows the code style of this project.

Current Behavior or Linked Issues

Background:

  • 2 years ago default value was set to 20 as part of MVP commit.
  • 1 year ago default value was changed from 20 to 30 as it is both Autoscaling requirement and MS docs requirement (see details below)
  • 2 weeks ago default value was set to 20 again (@heba92 can add details)

The same change was already approved and merged 1 year ago in MR !712 (merged) as part of autoscaling activities. Purpose of this MR is to revert recent change (20) to the same value as in MR !712 (merged) (30).

Does this introduce a breaking change?

  • [YES/NO]
    • It is NOT a breaking change for the environments deployed for the last 1 year or for the environments where max_pod value was overridden in terraform.tfvars file before deployment.
    • It can be a breaking change for the environments that were deployed more that 1 year ago (when default value was set to 20). @heba92 please let us know if you aware about such environments on your side. In this case there are two options how to avoid breaking change:
      • Migrate to new nodepool as described in MS docs
      • Override the value in terraform.tfvars

Other information

Reasons to set 30 as a default value:

  • MS doc recommendation: 30 is the default value recommended by MS docs: https://docs.microsoft.com/en-us/azure/aks/configure-azure-cni
  • MS doc requirement: All "system" AKS nodepools must have max_pod value set at minimum to 30 according to the next MS docs: https://docs.microsoft.com/en-us/azure/aks/use-system-pools?tabs=azure-cli#system-and-user-node-pools

    "System node pools must support at least 30 pods as described by the minimum and maximum value formula for pods."

  • Autoscaling requirement: we have at least 10 daemonset pods on each VM (in kube-system namespace) and in case max_pod is set to 20 those daemonsets are consuming half of that limit (10 pods from 20) which means we are able to run max 10 service pods on particular VM - which looks as unreasonable limitation for autoscaling cost efficiency - because in AKS we expect scaling to be performed based on cpu/memory requests. When we have so small max_pod per VM limitation (max_pod = 20) - scaling will happen based on this limitation much sooner than based on memory/cpu requests - which means we will never consume full memory capacity (efficiently) on each VM (and basically pay for memory that is not used)
Edited by Vasyl Leskiv [SLB]

Merge request reports