Skip to content

GitLab

Open
Created by Piotr Szwed@pszwed

CI/CD Architecture Improvement: Terraform Kubernetes Provider resources should be managed with separate apply operations

As outlined in the official Terraform documentation:

The most reliable way to configure the Kubernetes provider is to ensure that the cluster itself and the Kubernetes provider resources can be managed with separate apply operations. Data-sources can be used to convey values between the two stages as needed.

(source: https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs)

There should be two separate Azure DevOps Pipelines:

  • one for Kubernetes (AKS) Cluster creation
  • one for deploying Kubernetes provider resources (namespaces, helm charts, etc..) on top of already existing AKS

Today there is only one ADO Pipeline triggering terraform apply step that creates AKS cluster and Kubernetes provider resources as a single Terraform action.

This approach is not in line with Terraform internal architecture and leads to Terraform dependency tree issues described in the following thread: https://github.com/hashicorp/terraform/issues/27601

The same issue occurs in the OSDU case after we destroy AKS cluster (eg: via Azure Portal User Interface):

[...]
2021-06-09T13:52:05.6505397Z [31m╷[0m[0m
2021-06-09T13:52:05.6506490Z [31m│[0m [0m[1m[31mError: [0m[0m[1mGet "http://localhost/api/v1/namespaces/osdu": dial tcp [::1]:80: connect: connection refused[0m
2021-06-09T13:52:05.6507392Z [31m│[0m [0m
2021-06-09T13:52:05.6508006Z [31m│[0m [0m[0m  with kubernetes_namespace.osdu[0],
2021-06-09T13:52:05.6509247Z [31m│[0m [0m  on config_map.tf line 24, in resource "kubernetes_namespace" "osdu":
2021-06-09T13:52:05.6510077Z [31m│[0m [0m  24: resource "kubernetes_namespace" "osdu" [4m{[0m[0m
2021-06-09T13:52:05.6510672Z [31m│[0m [0m
2021-06-09T13:52:05.6511157Z [31m╵[0m[0m
2021-06-09T13:52:05.6511666Z [31m╷[0m[0m
2021-06-09T13:52:05.6512555Z [31m│[0m [0m[1m[31mError: [0m[0m[1mGet "http://localhost/api/v1/namespaces/cert-manager": dial tcp [::1]:80: connect: connection refused[0m
2021-06-09T13:52:05.6513455Z [31m│[0m [0m
2021-06-09T13:52:05.6514047Z [31m│[0m [0m[0m  with kubernetes_namespace.certs,
2021-06-09T13:52:05.6514834Z [31m│[0m [0m  on helm_certs.tf line 26, in resource "kubernetes_namespace" "certs":
2021-06-09T13:52:05.6515659Z [31m│[0m [0m  26: resource "kubernetes_namespace" "certs" [4m{[0m[0m
[...]

As a result, Terraform is unable to restore its Desired State which is a basic principle for Infrastructure as a Code Platform like OSDU.

I suggest to split the current terraform apply ADO Pipeline (pipeline-service-resources.yml) into two separate steps/pipelines as like suggested by Hashicorp and their official documentation so Terraform is always capable to restore its Desired State while running terraform apply.

Today explicit dependency rules like:

have no effect and are completely ignored by Terraform if triggered on already initialized/existing infrastructure!

Edited by Piotr Szwed