Merge branch 'pipeline_updates' into 'master'

Pipeline updates

See merge request osdu/platform/deployment-and-operations/infra-azure-provisioning!27

.gitlab/merge_request_templates/default.md

+## All Submissions:
+-------------------------------------
+* [YES/NO] Have you added an explanation of what your changes do and why you'd like us to include them?
+* [YES/NO] I have updated the documentation accordingly.
+* [YES/NO/NA] My code follows the code style of this project.
+
+
+## Current Behavior or Linked Issues
+-------------------------------------
+<!-- Please describe the current behavior that you are modifying, or link to a relevant issue. -->
+
+
+## Does this introduce a breaking change?
+-------------------------------------
+- [YES/NO]
+
+<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
+
+
+## Other information
+-------------------------------------
+<!-- Any other information that is important to this PR such as screenshots of how the component looks before and after the change. -->

.gitlab/merge_request_templates/infra_template.md

+## Infrastructure Submissions:
+-------------------------------------
+* [YES/NO] Have you added an explanation of what your changes do and why you'd like us to include them?
+* [YES/NO] I have updated the documentation accordingly.
+* [YES/NO/NA] I have added tests to cover my changes.
+* [YES/NO/NA] All new and existing tests passed.
+* [YES/NO/NA] I have formatted the terraform code.  _(`terraform fmt -recursive && go fmt ./...`)_
+
+## Current Behavior or Linked Issues
+-------------------------------------
+<!-- Please describe the current behavior that you are modifying, or link to a relevant issue. -->
+
+
+## Does this introduce a breaking change?
+-------------------------------------
+- [YES/NO]
+
+<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
+
+
+## Other information
+-------------------------------------
+<!-- Any other information that is important to this PR such as screenshots of how the component looks before and after the change. -->

README.md

@@ -99,6 +99,13 @@ export GIT_REPO=git@ssh.dev.azure.com:v3/${ADO_ORGANIZATION}/${ADO_PROJECT}/k8-g
   && git push -u origin --all)
 ```
 
+In order for Automated Pipelines to be able to work with this repository the following Permissions must be set in the ADO Project for `All Repositories/Permissions` on the user `osdu-mvp Build Service`.
+
+- Create Branch `Allow`
+- Contribute `Allow`
+- Contribute to Pull requests `Allow`
+
+
 
 ## Provision the Common Resources
 

docs/infra-automation.md

@@ -137,7 +137,7 @@ az pipelines create \
   --repository infra-azure-provisioning  \
   --branch master  \
   --repository-type tfsgit  \
-  --yaml-path /infra/templates/osdu-r3-mvp/pipeline-central-resources.yml  \
+  --yaml-path /devops/pipelines/infrastructure-central-resources.yml  \
   -ojson
 ```
 
@@ -153,7 +153,7 @@ az pipelines create \
   --repository infra-azure-provisioning  \
   --branch master  \
   --repository-type tfsgit  \
-  --yaml-path /infra/templates/osdu-r3-mvp/pipeline-data-partition.yml  \
+  --yaml-path /devops/pipelines/infrastructure-data-partition.yml  \
   -ojson
 ```
 
@@ -169,6 +169,6 @@ az pipelines create \
   --repository infra-azure-provisioning  \
   --branch master  \
   --repository-type tfsgit  \
-  --yaml-path /infra/templates/osdu-r3-mvp/pipeline-service-resources.yml  \
+  --yaml-path /devops/pipelines/infrastructure-service-resources.yml  \
   -ojson
 ```

docs/service-automation.md

@@ -298,7 +298,7 @@ Create the pipelines and run things in this exact order.
 1. Add a Pipeline for __chart-osdu-common__ to deploy common components.
 
     _Repo:_ `infra-azure-provisioning`
-    _Path:_ `/charts/osdu-common/pipeline.yml`
+    _Path:_ `/devops/pipelines/chart-osdu-common.yml`
     _Validate:_ https://<your_dns_name> is alive.
 
 ```bash
@@ -307,27 +307,42 @@ az pipelines create \
   --repository infra-azure-provisioning  \
   --branch master  \
   --repository-type tfsgit  \
-  --yaml-path /charts/osdu-common/pipeline.yml  \
+  --yaml-path /devops/pipelines/chart-osdu-common.yml  \
   -ojson
 ```
 
 2. Add a Pipeline for __chart-osdu-istio__  to deploy Istio components.
 
     _Repo:_ `infra-azure-provisioning`
-
-    _Path:_ `/charts/osdu-istio/pipeline.yml`
-
+    _Path:_ `/devops/pipelines/chart-osdu-istio.yml`
     _Validate:_ Pods are running in Istio Namespace.
 
+```bash
+az pipelines create \
+  --name 'chart-osdu-istio'  \
+  --repository infra-azure-provisioning  \
+  --branch master  \
+  --repository-type tfsgit  \
+  --yaml-path /devops/pipelines/chart-osdu-istio.yml  \
+  -ojson
+```
+
 
 3. Add a Pipeline for __chart-osdu-istio-auth__  to deploy Istio Authorization Policies.
 
     _Repo:_ `infra-azure-provisioning`
-
-    _Path:_ `/charts/osdu-istio-auth/pipeline.yml`
-
+    _Path:_ `/devops/pipelines/chart-osdu-istio-auth.yml`
     _Validate:_ Authorization Policies exist in osdu namespace.
 
+```bash
+az pipelines create \
+  --name 'chart-osdu-istio-auth'  \
+  --repository infra-azure-provisioning  \
+  --branch master  \
+  --repository-type tfsgit  \
+  --yaml-path /devops/pipelines/chart-osdu-istio-auth.yml  \
+  -ojson
+```
 
 __Create the Service Pipelines__
 
@@ -336,63 +351,115 @@ Create the pipelines and run things in this exact order.
 1. Add a Pipeline for __service-partition__  to deploy the Partition Service.
 
     _Repo:_ `partition`
-
     _Path:_ `/devops/azure/pipeline.yml`
-
     _Validate:_ https://<your_dns_name>/api/partition/v1/swagger-ui.html is alive.
 
+```bash
+az pipelines create \
+  --name 'service-partition'  \
+  --repository partition  \
+  --branch master  \
+  --repository-type tfsgit  \
+  --yaml-path /devops/azure/pipeline.yml  \
+  -ojson
+```
+
 
 2. Add a Pipeline for __service-entitlements-azure__  to deploy the Entitlements Service.
     > This pipeline may have to be run twice for integration tests to pass due to a preload data issue.
 
     _Repo:_ `entitlements-azure`
-
     _Path:_ `/devops/azure/pipeline.yml`
-
     _Validate:_ https://<your_dns_name>/entitlements/v1/swagger-ui.html is alive.
 
+```bash
+az pipelines create \
+  --name 'service-entitlements-azure'  \
+  --repository entitlements-azure  \
+  --branch master  \
+  --repository-type tfsgit  \
+  --yaml-path /devops/azure/pipeline.yml  \
+  -ojson
+```
+
 
 3. Add a Pipeline for __service-legal__  to deploy the Legal Service.
 
     _Repo:_ `legal`
-
     _Path:_ `/devops/azure/pipeline.yml`
-
     _Validate:_ https://<your_dns_name>/api/legal/v1/swagger-ui.html is alive.
 
+```bash
+az pipelines create \
+  --name 'service-legal'  \
+  --repository legal  \
+  --branch master  \
+  --repository-type tfsgit  \
+  --yaml-path /devops/azure/pipeline.yml  \
+  -ojson
+```
 
 4. Add a Pipeline for __service-storage__  to deploy the Storage Service.
 
     _Repo:_ `storage`
-
     _Path:_ `/devops/azure/pipeline.yml`
-
     _Validate:_ https://<your_dns_name>/api/storage/v2/swagger-ui.html is alive.
 
+```bash
+az pipelines create \
+  --name 'service-storage'  \
+  --repository storage  \
+  --branch master  \
+  --repository-type tfsgit  \
+  --yaml-path /devops/azure/pipeline.yml  \
+  -ojson
+```
+
 
 5. Add a Pipeline for __service-indexer-queue__  to deploy the Indexer Queue Function.
 
     _Repo:_ `indexer-queue`
-
     _Path:_ `/devops/azure/pipeline.yml`
-
     _Validate:_ ScaledObject exist in osdu namespace.
 
+```bash
+az pipelines create \
+  --name 'service-indexer-queue'  \
+  --repository indexer-queue  \
+  --branch master  \
+  --repository-type tfsgit  \
+  --yaml-path /devops/azure/pipeline.yml  \
+  -ojson
+```
 
 6. Add a Pipeline for __service-indexer__  to deploy the Indexer Service.
 
     _Repo:_ `indexer-service`
-
     _Path:_ `/devops/azure/pipeline.yml`
-
     _Validate:_ https://<your_dns_name>/api/indexer/v2/swagger-ui.html is alive.
 
+```bash
+az pipelines create \
+  --name 'service-indexer'  \
+  --repository indexer-service  \
+  --branch master  \
+  --repository-type tfsgit  \
+  --yaml-path /devops/azure/pipeline.yml  \
+  -ojson
+```
 
 7. Add a Pipeline for __service-search__  to deploy the Search Service.
 
     _Repo:_ `search-service`
-
     _Path:_ `/devops/azure/pipeline.yml`
-
     _Validate:_ https://<your_dns_name>/api/search/v2/swagger-ui.html is alive.
 
+```bash
+az pipelines create \
+  --name 'service-search'  \
+  --repository search-service  \
+  --branch master  \
+  --repository-type tfsgit  \
+  --yaml-path /devops/azure/pipeline.yml  \
+  -ojson
+```

infra/common_prepare.sh

@@ -546,7 +546,7 @@ export TF_VAR_remote_state_container="remote-state-container"
 export TF_VAR_resource_group_location="${AZURE_LOCATION}"
 export TF_VAR_cosmosdb_replica_location="${AZURE_PAIR_LOCATION}"
 
-export TF_VAR_central_resources_workspace_name="${UNIQUE}-cr"
+export TF_VAR_central_resources_workspace_name="cr-${UNIQUE}"
 
 export TF_VAR_principal_appId="$(az keyvault secret show --id https://$AZURE_VAULT.vault.azure.net/secrets/osdu-mvp-${UNIQUE}-principal-id --query value -otsv)"
 export TF_VAR_principal_name="osdu-mvp-${UNIQUE}-principal"

infra/templates/osdu-r3-mvp/central_resources/README.md

@@ -91,7 +91,7 @@ terraform init -backend-config "storage_account_name=${TF_VAR_remote_state_accou
 
 # This command configures terraform to use a workspace unique to you. This allows you to work
 # without stepping over your teammate's deployments
-TF_WORKSPACE="${UNIQUE}-cr"
+TF_WORKSPACE="cr-${UNIQUE}"
 terraform workspace new $TF_WORKSPACE || terraform workspace select $TF_WORKSPACE
 ```
 

infra/templates/osdu-r3-mvp/data_partition/README.md

@@ -59,7 +59,7 @@ terraform init -backend-config "storage_account_name=${TF_VAR_remote_state_accou
 
 # This command configures terraform to use a workspace unique to you. This allows you to work
 # without stepping over your teammate's deployments
-TF_WORKSPACE="${UNIQUE}-dp"
+TF_WORKSPACE="dp1-${UNIQUE}"
 terraform workspace new $TF_WORKSPACE || terraform workspace select $TF_WORKSPACE
 ```
 

infra/templates/osdu-r3-mvp/pipeline-central-resources.yml

@@ -65,9 +65,13 @@ stages:
 - template: /devops/infra-prepare-stage.yml
   parameters:
     environments:
-    - name: 'demo'
+    - name: 'dev'
       enablePrIsolation: false
       resourceNameIsolationLevel: 8
+    - ${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
+      - name: 'glab'
+        enablePrIsolation: false
+        resourceNameIsolationLevel: 8
 
     configurations:
     - jobName: central_resources

infra/templates/osdu-r3-mvp/pipeline-data-partition.yml

@@ -64,9 +64,13 @@ stages:
 - template: /devops/infra-prepare-stage.yml
   parameters:
     environments:
-    - name: 'demo'
+    - name: 'dev'
       enablePrIsolation: false
       resourceNameIsolationLevel: 8
+    - ${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
+      - name: 'glab'
+        enablePrIsolation: false
+        resourceNameIsolationLevel: 8
 
     configurations:
     - jobName: data_partition_1

infra/templates/osdu-r3-mvp/pipeline-service-resources.yml

@@ -63,9 +63,13 @@ stages:
 - template: /devops/infra-prepare-stage.yml
   parameters:
     environments:
-    - name: 'demo'
+    - name: 'dev'
       enablePrIsolation: false
       resourceNameIsolationLevel: 8
+    - ${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
+      - name: 'glab'
+        enablePrIsolation: false
+        resourceNameIsolationLevel: 8
 
     configurations:
     - jobName: service_resources

infra/templates/osdu-r3-mvp/service_resources/README.md

@@ -58,7 +58,7 @@ terraform init -backend-config "storage_account_name=${TF_VAR_remote_state_accou
 
 # This command configures terraform to use a workspace unique to you. This allows you to work
 # without stepping over your teammate's deployments
-TF_WORKSPACE="${UNIQUE}-sr"
+TF_WORKSPACE="sr-${UNIQUE}"
 terraform workspace new $TF_WORKSPACE || terraform workspace select $TF_WORKSPACE
 ```