Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in / Register
  • I infra-azure-provisioning
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 62
    • Issues 62
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 10
    • Merge requests 10
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Open Subsurface Data Universe SoftwareOpen Subsurface Data Universe Software
  • Platform
  • Deployment and Operations
  • infra-azure-provisioning
  • Issues
  • #172
Closed
Open
Issue created Jun 09, 2021 by Piotr Szwed@pszwed

CI/CD Architecture Improvement: Terraform Kubernetes Provider resources should be managed with separate apply operations

As outlined in the official Terraform documentation:

The most reliable way to configure the Kubernetes provider is to ensure that the cluster itself and the Kubernetes provider resources can be managed with separate apply operations. Data-sources can be used to convey values between the two stages as needed.

(source: https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs)

There should be two separate Azure DevOps Pipelines:

  • one for Kubernetes (AKS) Cluster creation
  • one for deploying Kubernetes provider resources (namespaces, helm charts, etc..) on top of already existing AKS

Today there is only one ADO Pipeline triggering terraform apply step that creates AKS cluster and Kubernetes provider resources as a single Terraform action.

This approach is not in line with Terraform internal architecture and leads to Terraform dependency tree issues described in the following thread: https://github.com/hashicorp/terraform/issues/27601

The same issue occurs in the OSDU case after we destroy AKS cluster (eg: via Azure Portal User Interface):

[...]
2021-06-09T13:52:05.6505397Z [31m╷[0m[0m
2021-06-09T13:52:05.6506490Z [31m│[0m [0m[1m[31mError: [0m[0m[1mGet "http://localhost/api/v1/namespaces/osdu": dial tcp [::1]:80: connect: connection refused[0m
2021-06-09T13:52:05.6507392Z [31m│[0m [0m
2021-06-09T13:52:05.6508006Z [31m│[0m [0m[0m  with kubernetes_namespace.osdu[0],
2021-06-09T13:52:05.6509247Z [31m│[0m [0m  on config_map.tf line 24, in resource "kubernetes_namespace" "osdu":
2021-06-09T13:52:05.6510077Z [31m│[0m [0m  24: resource "kubernetes_namespace" "osdu" [4m{[0m[0m
2021-06-09T13:52:05.6510672Z [31m│[0m [0m
2021-06-09T13:52:05.6511157Z [31m╵[0m[0m
2021-06-09T13:52:05.6511666Z [31m╷[0m[0m
2021-06-09T13:52:05.6512555Z [31m│[0m [0m[1m[31mError: [0m[0m[1mGet "http://localhost/api/v1/namespaces/cert-manager": dial tcp [::1]:80: connect: connection refused[0m
2021-06-09T13:52:05.6513455Z [31m│[0m [0m
2021-06-09T13:52:05.6514047Z [31m│[0m [0m[0m  with kubernetes_namespace.certs,
2021-06-09T13:52:05.6514834Z [31m│[0m [0m  on helm_certs.tf line 26, in resource "kubernetes_namespace" "certs":
2021-06-09T13:52:05.6515659Z [31m│[0m [0m  26: resource "kubernetes_namespace" "certs" [4m{[0m[0m
[...]

As a result, Terraform is unable to restore its Desired State which is a basic principle for Infrastructure as a Code Platform like OSDU.

I suggest to split the current terraform apply ADO Pipeline (pipeline-service-resources.yml) into two separate steps/pipelines as like suggested by Hashicorp and their official documentation so Terraform is always capable to restore its Desired State while running terraform apply.

Today explicit dependency rules like:

  • https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/infra/templates/osdu-r3-mvp/service_resources/helm_flux.tf#L34
  • https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/infra/templates/osdu-r3-mvp/service_resources/helm_flux.tf#L51
  • https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/infra/templates/osdu-r3-mvp/service_resources/helm_flux.tf#L108
  • https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/infra/templates/osdu-r3-mvp/service_resources/helm_keda.tf#L31
  • https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/infra/templates/osdu-r3-mvp/service_resources/helm_keda.tf#L41

have no effect and are completely ignored by Terraform if triggered on already initialized/existing infrastructure!

Edited Jun 09, 2021 by Piotr Szwed
Assignee
Assign to
Time tracking