GitLab

Context

Each service pod running in osdu namespace should have resource (i.e., cpu and memory) requests/limits defined. It is a best practice guidance as it helps kubernetes to schedule the pods according to their resource requirements and aid in better resource management.

Issue

In a multi micro-services system, like OSDU, it is difficult to incorporate resource limits across all micro-services. Services can miss out to have resource limits or can either specify too small or large resource limits. This can cause some services to use more resources than their fare quota and some services to starve for resources.

Solution

Default requests/limits for compute resources can be set at namespace level. This will ensure that all container created in that namespace will pick those default values for compute resources if they does not specify there own values. Services (or containers/pods) which needs different settings for compute resources can override the default namespace level settings by specifying requests/limits values in their deployment.yaml.

High Level Design

Introduction

This document explains how the default resource (i.e., CPU and memory) requests and limits can be set at a namespace level. Any container created in a namespace, that has default resource requests and limits set, will be assigned those default values if that container does not specify its own values.

LimitRange

Kubernetes’s LimitRange object can be used to specify default cpu/memory requests and limits for a namespace. Below is a sample configuration file for a LimitRange object.

apiVersion: v1 
kind: LimitRange 
metadata: 
  name: resource-limit-range 
  namespace: osdu 
spec: 
  limits: 
  - default: 
      memory: 2Gi 
      cpu: 1 
    defaultRequest: 
      memory: 2Gi 
      cpu: 0.5 
    type: Container 

Above configuration will set default cpu and memory requests as 0.5 and 2Gi respectively. Also, it will set the default cpu and memory limits to 1 and 2Gi respectively. These default values will be applicable for osdu namespace.

Any container created in osdu namespace will get these default settings for cpu and memory if that container doesn’t specify its own cpu/memory requests and limits. A container can override the default namespace level resource requests and limits by specifying its own values for the same.

Motivation

It is a best practice to set resource (i.e., CPU/memory) requests and limits for every pod in AKS cluster. But in a multi micro-services system, it is difficult to incorporate resource limits across all micro-services. Services can miss out to have resource limits or can either specify too small or large resource limits. This can cause some services to use more resources than their fare quota and some services to starve for resources.

Thus, specifying resource requests and limits at namespace level will guarantee the default share of resources to services (or pods) which don’t explicitly define resource limits. Services (or pods) which are fine-tuned with different set of requirements can specify their own requests and limits values which will override the namespace level settings.

Namespace limits in OSDU

Above figure describes the architecture of OSDU infra with namespace limits applied. All the OSDU services are deployed in osdu namespace. This osdu namespace will be configured with default settings for resource requests and limits by using LimitRange.

Services will pick up these default settings (e.g., Service1 and Service2). Services which have different requirements for resources can explicitly mention requests and limits values in their deployment.yaml (e.g., Service3). The values specified in deployment.yaml will override the namespace settings.

Potential Blockers

The default resource requests and limits set in a LimitRange object will be applicable for all the containers created in the specified namespace. If a service’s pod contains multiple containers, then all those containers will have the default settings.

Different containers within a pod can have different requirements for resources. Therefore, finding a common default values for resource requests and limits among different types of containers may be a difficult task.

NOTE: As of now only two types of containers are present in every pod in osdu namespace. One is istio-proxy and other is the service’s container. Resource requests and limits values for every istio-proxy container can be configured through common configuration file. Whereas default for service containers can be configured through LimitRange.

References

https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/

https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/

https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/cpu-default-namespace/