Merge branch 'release/0.11' into 'master'

Release/0.11

See merge request !486

CHANGELOG.md

@@ -2,6 +2,22 @@
 
 # Current Master
 
+# v0.11.0 (2021-9-1)
+__Infra Changes__
+- [Keda Upgrade to 2.x](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/128) - Follow this Documentation to do it: [Keda Upgrade](docs/keda-upgrade.md)
+- [BYOAD Enable](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/197)
+
+__Service Onboarded__
+- [Dataset Service](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/153)
+
+__Feature Changes__
+- [Add multi-partition support in Indexer Queue](https://community.opengroup.org/osdu/platform/system/indexer-queue/-/issues/6)
+- [Notification Service to use Service Bus](https://community.opengroup.org/osdu/platform/system/notification/-/issues/25)
+- [Notification loss, if migration not done](https://community.opengroup.org/osdu/platform/system/notification/-/blob/master/provider/notification-azure/docs/MIGRATION.md)
+
+__Deprecation Notes__
+- The Notification service has started functioning on Service Bus as the underlying PubSub. Starting v0.12.0, Event Grid will be deprecated. In v0.11.0, the service will have two deployments - one using Event Grid and one uses Service Bus.
+
 # v0.10.0 (2021-8-8)
 
 __Infra Changes__

README.md

@@ -174,24 +174,13 @@ __Installed Azure Resources__
 1. Resource Group
 2. Storage Account
 3. Key Vault
-4. A principal to be used by Terraform to create all resources for an OSDU Environment. _(Requires Grant Admin Approval)_
-5. A principal required by an OSDU environment deployment that will have root level access to that environment. _(Requires Grant Admin Approval)_
-6. An AD application to be leveraged in the future that defines and controls access to the OSDU Environment for AD Identity. _(future)_
+4. A principal to be used by Terraform to create all resources for an OSDU Environment.
+5. A principal required by an OSDU environment deployment.
+6. An AD application to be leveraged that defines and controls access to the OSDU Environment for AD Identity.
 7. An AD application to be used for negative integration testing
 
 > Removal would require deletion of all AD elements `osdu-mvp-{UNIQUE}-*`, unlocking and deleting the resource group.
 
-
-__Azure AD Admin Consent__
-
-2 service principals have been created that need to have an AD Admin `grant admin consent` on.
-
-1. osdu-mvp-{UNIQUE}-terraform  _(Azure AD Application Graph - Application.ReadWrite.OwnedBy)_
-2. osdu-mvp-{UNIQUE}-principal _(Microsoft Graph - Directory.Read.All)_
-
-For more information on Azure identity and authorization, see the official Microsoft documentation [here](https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-permissions-and-consent).
-
-
 ## Elastic Search Setup
 
 Infrastructure requires a bring your own Elastic Search Instance of a version of 7.x (ie: 7.11.1) with a valid https endpoint and the access information must now be stored in the Common KeyVault. The recommended method of Elastic Search is to use the [Elastic Cloud Managed Service from the Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/elastic.ec-azure?tab=Overview).

docs/byoad-enable.md

+# Enable BYOAD
+
+We've added a feature flag (aad_client_id) to enable or disable auto-creation of ad-application in central resources.
+
+# Updating existing infra to have custom AD Application
+Doing this will make current auth and refresh codes invalid. They'll need to be generated again.
+1. Users with manual deployment, if already not set, set aad_client_id = {{application client id of the custom ad application created}} in custom values file for terraform apply in central resources.
+2. Users with automated pipeline -
+   1. . Go to Pipelines Library in ADO
+   2. Go to `Infrastructure Pipeline Variables - demo` variable group
+   3. Add or update the below variable if already not set
+
+    | Variable | Value |
+      |----------|-------|
+    | TF_VAR_aad_client_id | {{application client id of manually created ad application}} |
+
+3. Users with automated pipeline should now run chart chart-osdu-istio and chart-osdu-istio-auth pipeline.
+4. Users with manual deployment need to re-install osdu-istio helm chart with new app-id.
+5. Delete all pods in the portal AKS. This will trigger a restart of all pods. Complete steps 5,6 and 7 in quick procession.
+6. While all pods are getting restarted, move to configuration in portal AKS. Select secrets tab and choose osdu/osdu-azure namespace.
+7. Delete active-directory from the results. This will trigger its recreation.
+8. Delete all pods again to make sure that new pods are using new active directory secrets.
+9. Run this script with required values substituted - [subscriberCreationRegisterService](./Trouble%20Shooting%20Guides/tsg-scripts/subscriberCreationRegisterService.ps1)
\ No newline at end of file

docs/faq.md

@@ -34,10 +34,6 @@ This is likely an issue with the `Application.ReadWrite.OwnedBy` permissions tha
 
 The common_prepare.sh script is a helper script that helps to perform the activities necessary to provision OSDU on Azure.  These activities can all be performed manually if desired.  Service Principals are created using the command `az ad sp create-for-rbac` which requires Owner permissions on a subscription to perform.
  
-## Why does the Service Principal used by Terraform to create an OSDU Environment Stamp require Azure AD Graph API access levels of `Application.ReadWrite.OwnedBy`?
-
-Terraform is used to provision an OSDU Environment Stamp a Service Principal is the identity used by Terraform to perform this action.  An OSDU Environment Stamp requires an AD Application used for Identity Management which is currently created by the Terraform Scripts.  In order for a Service Principal to be able to create Applications in AD, the permission of `Application.ReadWrite.OwnedBy` is required for the Azure AD Graph API.
- 
 ## Why does the Service Principal used internally within an OSDU Environment Stamp require MS Graph API  access levels of `Directory.ReadAll`?
 
 The OSDU Entitlement Service integrates with Azure AD.  The defined API spec for the service includes a Create method for which input criteria includes an email address.  This email address is looked up in Azure AD to confirm it exists and retrieve the Object Id of the user to be used as the source of identity which requires the permission of `Directory.ReadAll` for the MS Graph API.
@@ -59,7 +55,7 @@ This key pair can be used to ssh into an AKS node if needed.
 * osdu-mvp-xxx-terraform – A principal identity that can be used by Terraform for creating OSDU Resources
 * osdu-mvp-xxx-principal – A principal identity that is fed to an OSDU Deployment to be used as the Root Level Identity for that OSDU Environment
 * osdu-mvp-xxx-noaccess – A negative testing principal identity.
-* osdu-mvp-xxx-application – An AD Application for future use.  (Not currently used yet.)
+* osdu-mvp-xxx-application – An AD Application.
 
 ## What AAD Items are created by the central resource template?
 * osdu-mvp-crxxx-xxxx-app – An AD application that defines the OSDU Environment created.

docs/infra-automation.md

@@ -105,6 +105,7 @@ az pipelines variable-group create \
   TF_VAR_principal_password="${TF_VAR_principal_password}" \
   TF_VAR_resource_group_location="${REGION}" \
   TF_VAR_deploy_dp_airflow="false" \
+  TF_VAR_aad_client_id="$TF_VAR_application_clientid" \
   -ojson
 ```
 
@@ -121,6 +122,17 @@ To enable airflow multi partition turn on the feature flag by following the belo
   | TF_VAR_deploy_dp_airflow | true |
   | TF_VAR_ssl_challenge_required | true (if not using BYOC) <br> false (if using BYOC) |
 
+__Enable BYOAD__
+To enable byoad, turn on the feature flag by following the below steps. If you don't want to create your own AD Application, you can skip it.
+
+1. Go to Pipelines Library in ADO
+2. Go to `Infrastructure Pipeline Variables - demo` variable group
+3. Add or update the below variable
+
+| Variable | Value |
+  |----------|-------|
+| TF_VAR_aad_client_id | {{application client id of manually created ad application}} |
+
 
 __Setup and Configure the ADO Library `Infrastructure Pipeline Secrets - demo`__
 > This should be linked Secrets from Azure Key Vault `osducommon<random>`
@@ -161,7 +173,6 @@ az pipelines create \
   -ojson
 ```
 
-
 2. `infrastructure-data-partition`
 
   > For the first run of the pipeline approvals will need to be made for the 2 secure files and the Service Connection.

docs/service-data.md

@@ -11,7 +11,7 @@ UNIQUE="<your_osdu_unique>"         # ie: demo
 AZURE_DNS_NAME="<your_osdu_fqdn>"   # ie: osdu-$UNIQUE.contoso.com
 DATA_PARTITION="<your_partition>"   # ie:opendes
 ACR_REGISTRY="<repository>"         # ie: msosdu.azurecr.io
-TAG="<app_version>"                 # ie: 0.10.0
+TAG="<app_version>"                 # ie: 0.11.0
 
 # This logs your local Azure CLI in using the configured service principal.
 az login --service-principal -u $ARM_CLIENT_ID -p $ARM_CLIENT_SECRET --tenant $ARM_TENANT_ID
@@ -42,7 +42,7 @@ UNIQUE="<your_osdu_unique>"         # ie: demo
 AZURE_DNS_NAME="<your_osdu_fqdn>"   # ie: osdu-$UNIQUE.contoso.com
 DATA_PARTITION="<your_partition>"   # ie:opendes
 ACR_REGISTRY="<repository>"         # ie: msosdu.azurecr.io
-TAG="<app_version>"                 # ie: 0.10.0
+TAG="<app_version>"                 # ie: 0.11.0
 
 # This logs your local Azure CLI in using the configured service principal.
 az login --service-principal -u $ARM_CLIENT_ID -p $ARM_CLIENT_SECRET --tenant $ARM_TENANT_ID
@@ -74,7 +74,7 @@ UNIQUE="<your_osdu_unique>"         # ie: demo
 DNS_HOST="<your_osdu_fqdn>"         # ie: osdu-$UNIQUE.contoso.com
 DATA_PARTITION="<your_partition>"   # ie:opendes
 ACR_REGISTRY="<repository>"         # ie: msosdu.azurecr.io
-TAG="<app_version>"                 # ie: 0.10.0
+TAG="<app_version>"                 # ie: 0.11.0
 
 # This logs your local Azure CLI in using the configured service principal.
 az login --service-principal -u $ARM_CLIENT_ID -p $ARM_CLIENT_SECRET --tenant $ARM_TENANT_ID
@@ -107,7 +107,7 @@ UNIQUE="<your_osdu_unique>"                    # ie: demo
 AZURE_DNS_NAME="<your_osdu_fqdn>"              # ie: osdu-$UNIQUE.contoso.com
 DATA_PARTITION="<your_partition>"              # ie:opendes
 ACR_REGISTRY="<your_acr_fqdn>"                 # ie: myacr.azurecr.io
-TAG="<app_version>"                            # ie: 0.10.0
+TAG="<app_version>"                            # ie: 0.11.0
 DAG_TASK_IMAGE="segy-to-zgy-conversion-dag"    # i.e. name for the image in ACR
 AZURE_TENANT_ID="<azure tenant>"
 

infra/modules/providers/azure/ad-application/main.tf

@@ -13,12 +13,12 @@
 //  limitations under the License.
 
 data "azuread_service_principal" "main" {
-  count        = length(local.api_names)
+  count        = var.aad_client_id != "" ? 0 : length(local.api_names)
   display_name = local.api_names[count.index]
 }
 
 resource "azuread_application" "main" {
-  count                      = var.enable_bring_your_own_ad_app ? 0 : 1
+  count                      = var.aad_client_id != "" ? 0 : 1
   name                       = var.name
   homepage                   = coalesce(var.homepage, local.homepage)
   identifier_uris            = local.identifier_uris
@@ -60,14 +60,14 @@ resource "azuread_application" "main" {
 }
 
 resource "random_password" "main" {
-  count   = !var.enable_bring_your_own_ad_app && var.password == "" ? 1 : 0
+  count   = var.aad_client_id == "" && var.password == "" ? 1 : 0
   length  = 32
   special = false
 }
 
 resource "azuread_application_password" "main" {
-  count                 = !var.enable_bring_your_own_ad_app && var.password != null ? 1 : 0
-  application_object_id = var.enable_bring_your_own_ad_app ? null : azuread_application.main[0].object_id
+  count                 = var.aad_client_id == "" && var.password != null ? 1 : 0
+  application_object_id = var.aad_client_id != "" ? null : azuread_application.main[0].object_id
 
   value             = coalesce(var.password, random_password.main[0].result)
   end_date          = local.end_date

infra/modules/providers/azure/ad-application/output.tf

@@ -13,22 +13,22 @@
 //  limitations under the License.
 
 output "name" {
-  value       = var.enable_bring_your_own_ad_app ? null : azuread_application.main[0].name
+  value       = var.aad_client_id != "" ? null : azuread_application.main[0].name
   description = "The display name of the application."
 }
 
 output "id" {
-  value       = var.enable_bring_your_own_ad_app ? null : azuread_application.main[0].application_id
+  value       = var.aad_client_id != "" ? null : azuread_application.main[0].application_id
   description = "The ID of the application."
 }
 
 output "object_id" {
-  value       = var.enable_bring_your_own_ad_app ? null : azuread_application.main[0].object_id
+  value       = var.aad_client_id != "" ? null : azuread_application.main[0].object_id
   description = "The object ID of the application."
 }
 
 output "roles" {
-  value = var.enable_bring_your_own_ad_app ? null : {
+  value = var.aad_client_id != "" ? null : {
     for r in azuread_application.main[0].app_role :
     r.display_name => {
       id          = r.id
@@ -42,7 +42,7 @@ output "roles" {
 }
 
 output "password" {
-  value       = var.enable_bring_your_own_ad_app ? null : azuread_application_password.main.0.value
+  value       = var.aad_client_id != "" ? null : azuread_application_password.main.0.value
   sensitive   = true
   description = "The password for the application."
 }

infra/modules/providers/azure/ad-application/variables.tf

@@ -12,10 +12,10 @@
 //  See the License for the specific language governing permissions and
 //  limitations under the License.
 
-variable "enable_bring_your_own_ad_app" {
-  description = "Feature flag for BYOA"
-  default     = false
-  type        = bool
+variable "aad_client_id" {
+  description = "Existing Application AppId."
+  type        = string
+  default     = ""
 }
 
 variable "name" {
@@ -120,7 +120,7 @@ locals {
     }
   }
 
-  required_resource_access = [
+  required_resource_access = var.aad_client_id != "" ? [] : [
     for a in local.api_permissions : {
       resource_app_id = local.service_principals[a.name].application_id
       resource_access = concat(

infra/modules/providers/azure/keyvault/main.tf

@@ -28,7 +28,6 @@ resource "azurerm_key_vault" "keyvault" {
   resource_group_name = data.azurerm_resource_group.kv.name
   tenant_id           = data.azurerm_client_config.current.tenant_id
 
-  soft_delete_enabled        = true
   soft_delete_retention_days = 90
   purge_protection_enabled   = false
 

infra/templates/osdu-r3-mvp/central_resources/README.md

@@ -100,10 +100,17 @@ terraform workspace new $TF_WORKSPACE || terraform workspace select $TF_WORKSPAC
 ```bash
 # File location : /infra-azure-provisioning/infra/templates/osdu-r3-mvp/central_resources
 cp terraform.tfvars custom.tfvars
+
+# Do not run following commands if you wish to use ad application created/managed by terraform and
+# you've used common_prepare.sh for initial setup. Also, it requires setting of AZURE_VAULT, ADO_PROJECT and UNIQUE env variables.
+# These commands pull aad client id from common keyvault for the ad application created by common_prepare.sh. This aad client id is then used in terraform env.
+# if you have created common infra manually without common_prepare.sh, then manually set aad_client_id = "your ad application client id" in custom.tfvars and do not run these commands.
+TF_VAR_application_clientid=$(az keyvault secret show --id https://$AZURE_VAULT.vault.azure.net/secrets/${ADO_PROJECT}-${UNIQUE}-application-clientid --query value -otsv)
+echo -e "aad_client_id = \"$TF_VAR_application_clientid\"" >> custom.tfvars
 ```
 
 Execute the following commands to orchestrate a deployment.
-
+If we want to enable BYOAD (Bring your own AD Application), please go through following wiki [byoad-enable](./../../../../docs/byoad-enable.md)
 
 ```bash
 # See what terraform will try to deploy without actually deploying

infra/templates/osdu-r3-mvp/central_resources/main.tf

@@ -323,7 +323,7 @@ module "ad_application" {
     }
   ]
 
-  enable_bring_your_own_ad_app = var.enable_bring_your_own_ad_app
+  aad_client_id = var.aad_client_id
 }
 
 

infra/templates/osdu-r3-mvp/central_resources/secrets.tf

@@ -155,7 +155,7 @@ resource "azurerm_key_vault_secret" "principal_object_id" {
 // Add Application Information to KV
 resource "azurerm_key_vault_secret" "application_id" {
   name         = "aad-client-id"
-  value        = var.enable_bring_your_own_ad_app ? null : module.ad_application.id
+  value        = var.aad_client_id != "" ? var.aad_client_id : module.ad_application.id
   key_vault_id = module.keyvault.keyvault_id
 }
 

infra/templates/osdu-r3-mvp/central_resources/variables.tf

@@ -40,10 +40,10 @@ variable "feature_flag" {
   }
 }
 
-variable "enable_bring_your_own_ad_app" {
-  description = "Feature flag for BYOA"
-  default     = false
-  type        = bool
+variable "aad_client_id" {
+  description = "Existing Application AppId."
+  type        = string
+  default     = ""
 }
 
 variable "randomization_level" {

infra/templates/osdu-r3-mvp/service_resources/README.md

@@ -43,7 +43,11 @@ resource_tags = {
 }
 
 # Storage Settings
-storage_shares = [ "airflowdags" ]
+storage_shares = [
+  "airflowdags",
+  "unit",
+  "crs",
+  "crs-conversion"]
 storage_queues = [ "airflowlogqueue" ]
 ```
 

tools/test_data/README.md

@@ -18,6 +18,10 @@ Login to Azure CLI using the OSDU Environment Service Principal.
 
 ```bash
 # This logs your local Azure CLI in using the configured service principal.
+ARM_CLIENT_ID="<arm client id>"
+ARM_CLIENT_SECRET="<arm client secret>"
+ARM_TENANT_ID="<tenant id>"
+
 az login --service-principal -u $ARM_CLIENT_ID -p $ARM_CLIENT_SECRET --tenant $ARM_TENANT_ID
 ```
 
@@ -26,6 +30,7 @@ __Upload Storage Blob Test Data__
 This [file](../tools/test_data/Legal_COO.json) needs to be loaded into the Data Partition Storage Account in the container  `legal-service-azure-configuration`.
 
 ```bash
+UNIQUE="<your_osdu_unique>" 
 GROUP=$(az group list --query "[?contains(name, 'cr${UNIQUE}')].name" -otsv)
 ENV_VAULT=$(az keyvault list --resource-group $GROUP --query [].name -otsv)
 PARTITION_NAME=opendes
@@ -65,6 +70,7 @@ These files need to be uploaded into the proper Cosmos Collections with the requ
 
 ```bash
 # Retrieve Values from Common Key Vault
+COMMON_VAULT="<common keyvault created in common prepare phase>"
 export NO_DATA_ACCESS_TESTER=$(az keyvault secret show --id https://$COMMON_VAULT.vault.azure.net/secrets/osdu-mvp-${UNIQUE}-noaccess-clientid --query value -otsv)
 
 # Retrieve Values from Environment Key Vault