Merge remote-tracking branch 'origin/master' into...

Merge remote-tracking branch 'origin/master' into users/kiveerap/SplitingProfileAndInstalltionIntoMultipleFiles

README.md

@@ -368,12 +368,14 @@ done
 | AZURE_AD_USER_EMAIL                           | `$(ad-user-email)`                        |
 | AZURE_AD_USER_OID                             | `$(ad-user-oid)`                          |
 | AZURE_LEGAL_TOPICNAME                         | `legaltags`                               |
+| DEPLOY_ENV                                    | `empty`                                   |
 | ENTITLEMENT_URL                               | `https://$(DNS_HOST)/entitlements/v1/`    |
 | EXPIRED_TOKEN                                 | <an_expired_token>                        |
 | HOST_URL                                      | `https://$(DNS_HOST)/`                    |
 | LEGAL_URL                                     | `https://$(DNS_HOST)/api/legal/v1/`       |
 | NO_DATA_ACCESS_TESTER                         | `$(osdu-infra-azg-test-app-noaccess-id)`  |
 | NO_DATA_ACCESS_TESTER_SERVICEPRINCIPAL_SECRET | `$(osdu-infra-azg-test-app-noaccess-key)` |
+| PUBSUB_TOKEN                                  | `empty`                                   |
 | SERVICE_CONNECTION_NAME                       | <your_service_connection_name>            |
 | GOOGLE_CLOUD_PROJECT                          | `opendes`                                 |
 
@@ -483,6 +485,7 @@ __- Setup and Configure the ADO Library__ `Azure Service Release - storage`
 
 
 
+
 __- Setup and Configure the ADO Library__ `Azure Service Release - indexer`
 
 | Variable | Value |
@@ -504,6 +507,14 @@ __- Setup and Configure the ADO Library__ `Azure Service Release - search`
 | SERVICE_RESOURCE_NAME | `$(AZURE_SEARCH_SERVICE_NAME)` |
 
 
+### Load Storage Container Integration Test Data
+
+The data to be loaded before services are deployed and can be found in the osdu-infrastructure repository `osdu-infrastructure/docs/osdu/integration-test-data/`.
+
+__Container: legal-service-azure-configuration__
+
+- Legal_COO.json
+
 
 ### Load Cosmos DB Integration Test Data
 
@@ -532,29 +543,77 @@ The data to be loaded before services are deployed and can be found in the osdu-
 ### Configure the ADO Charts and Service Pipelines
 
 Create the pipelines and run things in this exact order.
-> Ensure DNS is configured for your Gateway IP to DNS_HOST
 
-- Add a Pipeline __chart-osdu-common__ -->  Repo: infra-azure-provisioning Path:`/charts/osdu-common/pipeline.yml` and execute it.
-  - This pipeline will deploy to flux the common components which includes ingress.
-  - Validate the URL is alive.  https://<your_dns_name>
 
+1. Add a Pipeline for __chart-osdu-common__ to deploy common components.
+    > Ensure DNS is configured for your Gateway IP to DNS_HOST prior.
 
-- Add a Pipeline __service-partition__ -->  Repo: partition Path:`/devops/azure/pipeline.yml` and execute it.
-  - This pipeline will deploy to flux the partition service.
-  > Partition Service requires an execution call to set Partition Data Manually at this point.
+    _Repo:_ `infra-azure-provisioning`
 
+    _Path:_ `/charts/osdu-common/pipeline.yml`
 
-- Add a Pipeline __service-entitlements-azure__ -->  Repo: entitlements-azure Path:`/devops/azure/pipeline.yml` and execute it.
-  - This pipeline will have to be run twice for integration tests to pass due to a preload data issue.
+    _Validate:_ https://<your_dns_name> is alive.
 
 
-- Add a Pipeline __service-legal__ -->  Repo: legal Path:`/devops/legal/pipeline.yml` and execute it.
+2. Add a Pipeline for __chart-osdu-istio__  to deploy Istio components.
 
+    _Repo:_ `infra-azure-provisioning`
+
+    _Path:_ `/charts/osdu-istio/pipeline.yml`
+
+    _Validate:_ Pods are running in Istio Namespace.
+
+
+3. Add a Pipeline for __chart-osdu-istio-auth__  to deploy Istio Authorization Policies.
+
+    _Repo:_ `infra-azure-provisioning`
+
+    _Path:_ `/charts/osdu-istio-auth/pipeline.yml`
+
+    _Validate:_ Authorization Policies exist in osdu namespace.
+
+
+4. Add a Pipeline for __service-partition__  to deploy the Partition Service.
+
+    _Repo:_ `partition`
+
+    _Path:_ `/devops/azure/pipeline.yml`
+
+    _Validate:_ https://<your_dns_name>/api/partition/v1/swagger-ui.html is alive.
 
-- Add a Pipeline __indexer-queue__ -->  Repo: indexer-queue Path:`/devops/azure/pipeline.yml` and execute it.
 
 
-- Add a Pipeline __storage__ -->  Repo: storage Path:`/devops/azure/pipeline.yml` and execute it.
+5. Add a Pipeline for __service-entitlements-azure__  to deploy the Entitlements Service.
+    > This pipeline may have to be run twice for integration tests to pass due to a preload data issue.
+
+    _Repo:_ `entitlements-azure`
+
+    _Path:_ `/devops/azure/pipeline.yml`
+
+    _Validate:_ https://<your_dns_name>/entitlements/v1/swagger-ui.html is alive.
+
+
+6. Add a Pipeline for __service-legal__  to deploy the Legal Service.
+
+    _Repo:_ `legal`
+
+    _Path:_ `/devops/azure/pipeline.yml`
+
+    _Validate:_ https://<your_dns_name>/api/legal/v1/swagger-ui.html is alive.
+
+
+6. Add a Pipeline for __service-storage__  to deploy the Storage Service.
+
+    _Repo:_ `storage`
+
+    _Path:_ `/devops/azure/pipeline.yml`
+
+    _Validate:_ https://<your_dns_name>/api/storage/v2/swagger-ui.html is alive.
+
+
+
+
+- Add a Pipeline __indexer-queue__ -->  Repo: indexer-queue Path:`/devops/azure/pipeline.yml` and execute it.
 
 
 - Add a Pipeline __indexer__ -->  Repo: indexer Path:`/devops/azure/pipeline.yml` and execute it.

charts/helm-config.yaml

@@ -29,10 +29,10 @@ global:
     admin: #{ADMIN_EMAIL}#
     sslServer: https://acme-v02.api.letsencrypt.org/directory  # Production
 
-################################################################################
-# Specify the istio specific values
-# based64 encoded username and password
-#
-istio:
-  username: #{istio-username}#
-  password: #{istio-password}#
+  ################################################################################
+  # Specify the istio specific values
+  # based64 encoded username and password
+  #
+  istio:
+    username: #{istio-username}#
+    password: #{istio-password}#

charts/osdu-istio-auth/Chart.yaml

@@ -13,7 +13,7 @@
 #  limitations under the License.
 
 apiVersion: v2
-name: istio
+name: osdu-istio-auth
 appVersion: "latest"
 description: Helm Chart for installing istio auth on osdu services.
 version: 0.1.0

charts/osdu-istio-auth/pipeline.yml

+#  Copyright © Microsoft Corporation
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+# Required Group Variables - `Azure - OSDU`
+  # AGENT_POOL
+  # AZURE_TENANT_ID
+  # ENV_SUBSCRIPTION_ID
+  # ENV_APP_ID
+  # ENV_BASE_NAME_21
+  # ENV_IDENTITY_ID
+  # ENV_HOST
+  # ADMIN_EMAIL
+  # ENV_REGISTRY
+  # ISTIO_USERNAME
+  # ISTIO_PASSWORD
+
+  trigger:
+    batch: true
+    branches:
+      include:
+        - master
+    paths:
+      include:
+        - /charts/osdu-istio-auth/*
+      exclude:
+        - /**/*.md
+
+  pr:
+    autoCancel: false
+    branches:
+      include:
+        - "*"
+      exclude:
+        - master
+    paths:
+      include:
+        - /charts/osdu-istio-auth/*
+      exclude:
+        - /**/*.md
+
+  resources:
+    repositories:
+    - repository: FluxRepo
+      type: git
+      name: k8-gitops-manifests
+
+  variables:
+    - group: 'Azure - OSDU'
+    - group: 'Azure - OSDU secrets'
+
+    - name: serviceName
+      value: "osdu-istio-auth"
+    - name: chartPath
+      value: "charts/osdu-istio-auth"
+    - name: valuesFile
+      value: "charts/helm-config.yaml"
+    - name: 'MANIFEST_REPO'
+      value: $[ resources.repositories['FluxRepo'].name ]
+
+  stages:
+    - template: /devops/chart-stages.yml
+      parameters:
+        serviceName: ${{ variables.serviceName }}
+        chartPath: ${{ variables.chartPath }}
+        valuesFile: ${{ variables.valuesFile }}
+        skipDeploy: ${{ variables.SKIP_DEPLOY }}
+        skipCheck: true
+        providers:
+          - name: Azure
+            environments: ["dev"]

charts/osdu-istio/Chart.yaml

@@ -13,7 +13,7 @@
 #  limitations under the License.
 
 apiVersion: v2
-name: istio
+name: osdu-istio
 appVersion: "latest"
 description: Helm Chart for installing istio for osdu on azure.
 version: 0.1.0

charts/osdu-istio/pipeline.yml

+#  Copyright © Microsoft Corporation
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+# Required Group Variables - `Azure - OSDU`
+  # AGENT_POOL
+  # AZURE_TENANT_ID
+  # ENV_SUBSCRIPTION_ID
+  # ENV_APP_ID
+  # ENV_BASE_NAME_21
+  # ENV_IDENTITY_ID
+  # ENV_HOST
+  # ADMIN_EMAIL
+  # ENV_REGISTRY
+  # ISTIO_USERNAME
+  # ISTIO_PASSWORD
+
+  trigger:
+    batch: true
+    branches:
+      include:
+        - master
+    paths:
+      include:
+        - /charts/osdu-istio/*
+      exclude:
+        - /**/*.md
+
+  pr:
+    autoCancel: false
+    branches:
+      include:
+        - "*"
+      exclude:
+        - master
+    paths:
+      include:
+        - /charts/osdu-istio/*
+      exclude:
+        - /**/*.md
+
+  resources:
+    repositories:
+    - repository: FluxRepo
+      type: git
+      name: k8-gitops-manifests
+
+  variables:
+    - group: 'Azure - OSDU'
+    - group: 'Azure - OSDU secrets'
+
+    - name: serviceName
+      value: "osdu-istio"
+    - name: chartPath
+      value: "charts/osdu-istio"
+    - name: valuesFile
+      value: "charts/helm-config.yaml"
+    - name: 'MANIFEST_REPO'
+      value: $[ resources.repositories['FluxRepo'].name ]
+
+  stages:
+    - template: /devops/chart-stages.yml
+      parameters:
+        serviceName: ${{ variables.serviceName }}
+        chartPath: ${{ variables.chartPath }}
+        valuesFile: ${{ variables.valuesFile }}
+        skipDeploy: ${{ variables.SKIP_DEPLOY }}
+        skipCheck: true
+        providers:
+          - name: Azure
+            environments: ["dev"]

devops/chart-stages.yml

@@ -53,5 +53,6 @@ stages:
                 - template: tasks/flux-chart-wait.yml
                   parameters:
                     serviceName: ${{parameters.serviceName}}
+                    skipCheck: ${{ parameters.skipCheck }}
                     environment: ${{ environment }}
                     imageRepoName: '${{ parameters.serviceName }}'

devops/deploy-stages.yml

@@ -24,7 +24,7 @@ parameters:
   testCoreMavenOptions: ''
   testCoreMavenGoal: 'install'
   skipDeploy: false
-  skipTest: false
+  skipTest: 'false'
 
 stages:
 

devops/tasks/deployment-steps.yml

@@ -42,7 +42,7 @@ steps:
       environment: ${{ parameters.environment }}
       jarFilePath: $(DetectJarScript.JAR_FILE_PATH)
       providerName: ${{ parameters.providerName }}
-      imageRepoName: '${{ parameters.serviceName }}'
+      imageRepoName: ${{ parameters.serviceName }}
 
   - template: aks-deployment-steps.yml
     parameters:
@@ -83,7 +83,7 @@ steps:
 
   - task: Maven@3
     displayName: 'Maven run integration test'
-    condition: and(succeeded(), ne('${{ parameters.skipTest }}', false))
+    condition: and(succeeded(), eq('${{ parameters.skipTest }}', 'false'))
     inputs:
       mavenPomFile: '$(System.DefaultWorkingDirectory)/$(MAVEN_INTEGRATION_TEST_POM_FILE_PATH)'
       goals: ${{ parameters.integrationTestMavenGoal }}

devops/tasks/flux-chart-wait.yml

@@ -19,6 +19,7 @@ steps:
     env:
       SERVICE_NAME: ${{parameters.serviceName}}
       BASE_NAME_SR: $(base-name-sr)
+      SKIP_CHECK: ${{parameters.skipCheck}}
     inputs:
       azureSubscription: '$(SERVICE_CONNECTION_NAME)'
       addSpnToEnvironment: true
@@ -27,6 +28,14 @@ steps:
         #!/usr/bin/env bash
         set -euo pipefail
 
+        if [[ $SKIP_CHECK == true ]]
+        then
+            echo "***********************"
+            echo "FLUX DEPLOYMENT SKIPPED"
+            echo "***********************"
+            exit 0
+        fi
+
         echo "Logging in to AKS"
         echo "------------------------------------"
         sudo az aks install-cli

devops/tasks/helm-template.yml

@@ -29,11 +29,9 @@ steps:
         #!/usr/bin/env bash
 
           cd $(Build.SourcesDirectory)/$(Build.Repository.Name)
+
           mkdir $CHART_PATH/$GENERATION_PATH
           cat $(Build.SourcesDirectory)/$(Build.Repository.Name)/${{parameters.chartPath}}/values.yaml
 
-          # echo "Creating Helm Values File"
-          # envsubst < $VALUES_FILE > $CHART_PATH/values.yml
-
           echo "Extracting Manifest"
           helm template $SERVICE_NAME $CHART_PATH -f $(Build.SourcesDirectory)/$(Build.Repository.Name)/${{parameters.chartPath}}/values.yaml --output-dir $CHART_PATH/$GENERATION_PATH