Automation gaps helm package/install/publish Azure 231 - 1

Related to: osdu/platform/deployment-and-operations/infra-azure-provisioning#231 (closed)

Adapted azure.yml gitlab pipeline as follows:

• Use helm-charts from helm-charts-azure repository in Gitlab pipelines. - COVERED
• Service CI-CD pipeline will get helm-charts from the "helm-charts-azure" repository in the azure-containerize step. - COVERED
• The helm-package step should also update "app-version" with the corresponding tag used for docker image. - COVERED using commit sha on both
• Use helm upgrade command in "azure-deploy" step to push a new release. - COVERED and added retrocompatibility
• When the service's CI-CD pipeline is triggered against release branch or a tag, a "publish" stage should get executed, that would copy helm charts and docker images from OSDU Gitlab ACR to the "msosdu" acr. - DONE on first MR and this will include as well helm package version parsing.

Additions:

• azure_containerize_helm Stage it is added to get the helm charts from helm-charts-azure, package based on the AZURE_HELM_SUBDIR var, which is not required and it is optional, however, the AZURE_HELM_SUBDIR will be set as default osdu-azure/$CI_PROJECT_NAME, this might work with storage or secret, but not for indexer, or ingestion-workflow names.
  1. It will remove prefix "trusted-" from the branch which is running, if the branch exists in helm-chart-azure it will download that branch helm version of the helm-charts-azure.
     • Full Pipeline Example
     • Job Example existing branch and path
  2. If the branch does not exists, will download the default branch of the helm-charts-azure.
     • Pipeline example
     • Job Example / not exist branch nor path
  3. If the AZURE_HELM_SUBDIR directory I.E (osdu-azure/myservice) is not found, will fail and abort the helm package/acrupload
     • Job Example / not exist branch nor path
  4. If the AZURE_HELM_SUBDIR is found will package, update appVersion as well as helm chart version and upload to the AZURE_HELM_SUBDIR/service path in the acr, I.E: if AZURE_HELM_SUBDIR=osdu-azure/storage, it will upload in oci://acr.ms/osdu-azure/storage:patchedversion
  5. If pipeline detects that it is running on release or tag condition (checked through gitlab env vars), it will use same approach as azure-containerize stage, it will patch the appVersion, the helm version based on the tag/release and will push in oci://glab-acr.ms/release-<tag>/osdu-azure/storage:patchedversion, this will be a helm with parsed version.
     • Full Pipeline Example
     • Job Example existing branch and path and release
• azure_deploy refactor
  • Upgraded to helm 3.9 version and used azure-cli image from msft registry.
  • Added compatibility for helm upgrade with remote oci://acr.ms protocol (helm-charts-azure),
    1. If the publish stage failed (helm-chart-azure) for any reason, it will fallback to the /devops/azure/charts templates for retrocompatibility.
       • Job Example
    2. If the remote helm acr chart it is found, will be used to install service.
       • Job Example
       • Full Pipeline Example
  • Added removal of chart if found in default namespace (Migration will happen for all services eventually as all charts are installed in default namespace as for now). To avoid migrating all at once, this can be triggered programmatically and incrementally (to remove helm uninstall step).
• azure_publish
  • Move release pre-packaged helm charts to the msosdu ACR, based on the parser rules that we implemented.
    • Parse Rules: (remove "-service" sufix or "ingestion" prefix)
    • Parse Rules: (remove "v" from tag for versioning tagging)
    • Rename ddms: (Only for sdms and wdms)
    • Full Pipeline Example
    • Publish example on helm and image
  • Would like to use acr import instead of crane in last step, however, found multiple issues with subscriptions and stuff, therefore we opted as for now to use crane.