Automated byoad

README.md

@@ -174,24 +174,13 @@ __Installed Azure Resources__
 1. Resource Group
 2. Storage Account
 3. Key Vault
-4. A principal to be used by Terraform to create all resources for an OSDU Environment. _(Requires Grant Admin Approval)_
-5. A principal required by an OSDU environment deployment that will have root level access to that environment. _(Requires Grant Admin Approval)_
-6. An AD application to be leveraged in the future that defines and controls access to the OSDU Environment for AD Identity. _(future)_
+4. A principal to be used by Terraform to create all resources for an OSDU Environment.
+5. A principal required by an OSDU environment deployment.
+6. An AD application to be leveraged that defines and controls access to the OSDU Environment for AD Identity.
 7. An AD application to be used for negative integration testing
 
 > Removal would require deletion of all AD elements `osdu-mvp-{UNIQUE}-*`, unlocking and deleting the resource group.
 
-
-__Azure AD Admin Consent__
-
-2 service principals have been created that need to have an AD Admin `grant admin consent` on.
-
-1. osdu-mvp-{UNIQUE}-terraform  _(Azure AD Application Graph - Application.ReadWrite.OwnedBy)_
-2. osdu-mvp-{UNIQUE}-principal _(Microsoft Graph - Directory.Read.All)_
-
-For more information on Azure identity and authorization, see the official Microsoft documentation [here](https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-permissions-and-consent).
-
-
 ## Elastic Search Setup
 
 Infrastructure requires a bring your own Elastic Search Instance of a version of 7.x (ie: 7.11.1) with a valid https endpoint and the access information must now be stored in the Common KeyVault. The recommended method of Elastic Search is to use the [Elastic Cloud Managed Service from the Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/elastic.ec-azure?tab=Overview).

docs/byoad-enable.md

 # Enable BYOAD
 
-We've added a feature flag to enable or disable auto-creation of ad-application in central resources.
+We've added a feature flag (aad_client_id) to enable or disable auto-creation of ad-application in central resources.
 
 # Updating existing infra to have custom AD Application
 Doing this will make current auth and refresh codes invalid. They'll need to be generated again.
-1. Set enable_bring_your_own_ad_app=true in custom values file for terraform apply in central resources.
-2. Post success of terraform apply for central resources, add application id of custom ad application to aad-client-id key in central resources keyvault.
+1. Users with manual deployment, if already not set, set aad_client_id = {{application client id of the custom ad application created}} in custom values file for terraform apply in central resources.
+2. Users with automated pipeline -
+   1. . Go to Pipelines Library in ADO
+   2. Go to `Infrastructure Pipeline Variables - demo` variable group
+   3. Add or update the below variable if already not set
+
+    | Variable | Value |
+      |----------|-------|
+    | TF_VAR_aad_client_id | {{application client id of manually created ad application}} |
+
 3. Users with automated pipeline should now run chart chart-osdu-istio and chart-osdu-istio-auth pipeline.
 4. Users with manual deployment need to re-install osdu-istio helm chart with new app-id.
 5. Delete all pods in the portal AKS. This will trigger a restart of all pods. Complete steps 5,6 and 7 in quick procession.

docs/faq.md

@@ -34,10 +34,6 @@ This is likely an issue with the `Application.ReadWrite.OwnedBy` permissions tha
 
 The common_prepare.sh script is a helper script that helps to perform the activities necessary to provision OSDU on Azure.  These activities can all be performed manually if desired.  Service Principals are created using the command `az ad sp create-for-rbac` which requires Owner permissions on a subscription to perform.
  
-## Why does the Service Principal used by Terraform to create an OSDU Environment Stamp require Azure AD Graph API access levels of `Application.ReadWrite.OwnedBy`?
-
-Terraform is used to provision an OSDU Environment Stamp a Service Principal is the identity used by Terraform to perform this action.  An OSDU Environment Stamp requires an AD Application used for Identity Management which is currently created by the Terraform Scripts.  In order for a Service Principal to be able to create Applications in AD, the permission of `Application.ReadWrite.OwnedBy` is required for the Azure AD Graph API.
- 
 ## Why does the Service Principal used internally within an OSDU Environment Stamp require MS Graph API  access levels of `Directory.ReadAll`?
 
 The OSDU Entitlement Service integrates with Azure AD.  The defined API spec for the service includes a Create method for which input criteria includes an email address.  This email address is looked up in Azure AD to confirm it exists and retrieve the Object Id of the user to be used as the source of identity which requires the permission of `Directory.ReadAll` for the MS Graph API.
@@ -59,7 +55,7 @@ This key pair can be used to ssh into an AKS node if needed.
 * osdu-mvp-xxx-terraform – A principal identity that can be used by Terraform for creating OSDU Resources
 * osdu-mvp-xxx-principal – A principal identity that is fed to an OSDU Deployment to be used as the Root Level Identity for that OSDU Environment
 * osdu-mvp-xxx-noaccess – A negative testing principal identity.
-* osdu-mvp-xxx-application – An AD Application for future use.  (Not currently used yet.)
+* osdu-mvp-xxx-application – An AD Application.
 
 ## What AAD Items are created by the central resource template?
 * osdu-mvp-crxxx-xxxx-app – An AD application that defines the OSDU Environment created.

docs/infra-automation.md

@@ -105,6 +105,7 @@ az pipelines variable-group create \
   TF_VAR_principal_password="${TF_VAR_principal_password}" \
   TF_VAR_resource_group_location="${REGION}" \
   TF_VAR_deploy_dp_airflow="false" \
+  TF_VAR_aad_client_id="$TF_VAR_application_clientid" \
   -ojson
 ```
 
@@ -130,7 +131,7 @@ To enable byoad, turn on the feature flag by following the below steps. If you d
 
 | Variable | Value |
   |----------|-------|
-| TF_VAR_enable_bring_your_own_ad_app | true |
+| TF_VAR_aad_client_id | {{application client id of manually created ad application}} |
 
 
 __Setup and Configure the ADO Library `Infrastructure Pipeline Secrets - demo`__
@@ -172,10 +173,6 @@ az pipelines create \
   -ojson
 ```
 
-If you've enabled BYOAD, then following steps need to be done - 
-1. Post success of terraform apply for central resources, add application id of custom ad application to aad-client-id key in central resources keyvault.
-
-
 2. `infrastructure-data-partition`
 
   > For the first run of the pipeline approvals will need to be made for the 2 secure files and the Service Connection.

infra/modules/providers/azure/ad-application/main.tf

@@ -18,7 +18,7 @@ data "azuread_service_principal" "main" {
 }
 
 resource "azuread_application" "main" {
-  count                      = var.enable_bring_your_own_ad_app ? 0 : 1
+  count                      = var.aad_client_id != "" ? 0 : 1
   name                       = var.name
   homepage                   = coalesce(var.homepage, local.homepage)
   identifier_uris            = local.identifier_uris
@@ -60,14 +60,14 @@ resource "azuread_application" "main" {
 }
 
 resource "random_password" "main" {
-  count   = !var.enable_bring_your_own_ad_app && var.password == "" ? 1 : 0
+  count   = var.aad_client_id == "" && var.password == "" ? 1 : 0
   length  = 32
   special = false
 }
 
 resource "azuread_application_password" "main" {
-  count                 = !var.enable_bring_your_own_ad_app && var.password != null ? 1 : 0
-  application_object_id = var.enable_bring_your_own_ad_app ? null : azuread_application.main[0].object_id
+  count                 = var.aad_client_id == "" && var.password != null ? 1 : 0
+  application_object_id = var.aad_client_id != "" ? null : azuread_application.main[0].object_id
 
   value             = coalesce(var.password, random_password.main[0].result)
   end_date          = local.end_date

infra/modules/providers/azure/ad-application/output.tf

@@ -13,22 +13,22 @@
 //  limitations under the License.
 
 output "name" {
-  value       = var.enable_bring_your_own_ad_app ? null : azuread_application.main[0].name
+  value       = var.aad_client_id != "" ? null : azuread_application.main[0].name
   description = "The display name of the application."
 }
 
 output "id" {
-  value       = var.enable_bring_your_own_ad_app ? null : azuread_application.main[0].application_id
+  value       = var.aad_client_id != "" ? null : azuread_application.main[0].application_id
   description = "The ID of the application."
 }
 
 output "object_id" {
-  value       = var.enable_bring_your_own_ad_app ? null : azuread_application.main[0].object_id
+  value       = var.aad_client_id != "" ? null : azuread_application.main[0].object_id
   description = "The object ID of the application."
 }
 
 output "roles" {
-  value = var.enable_bring_your_own_ad_app ? null : {
+  value = var.aad_client_id != "" ? null : {
     for r in azuread_application.main[0].app_role :
     r.display_name => {
       id          = r.id
@@ -42,7 +42,7 @@ output "roles" {
 }
 
 output "password" {
-  value       = var.enable_bring_your_own_ad_app ? null : azuread_application_password.main.0.value
+  value       = var.aad_client_id != "" ? null : azuread_application_password.main.0.value
   sensitive   = true
   description = "The password for the application."
 }

infra/modules/providers/azure/ad-application/variables.tf

@@ -12,10 +12,10 @@
 //  See the License for the specific language governing permissions and
 //  limitations under the License.
 
-variable "enable_bring_your_own_ad_app" {
-  description = "Feature flag for BYOA"
-  default     = false
-  type        = bool
+variable "aad_client_id" {
+  description = "Existing Application AppId."
+  type        = string
+  default     = ""
 }
 
 variable "name" {

infra/templates/osdu-r3-mvp/central_resources/README.md

@@ -100,6 +100,13 @@ terraform workspace new $TF_WORKSPACE || terraform workspace select $TF_WORKSPAC
 ```bash
 # File location : /infra-azure-provisioning/infra/templates/osdu-r3-mvp/central_resources
 cp terraform.tfvars custom.tfvars
+
+# Do not run following commands if you wish to use ad application created/managed by terraform and
+# you've used common_prepare.sh for initial setup. Also, it requires setting of AZURE_VAULT, ADO_PROJECT and UNIQUE env variables.
+# These commands pull aad client id from common keyvault for the ad application created by common_prepare.sh. This aad client id is then used in terraform env.
+# if you have created common infra manually without common_prepare.sh, then manually set aad_client_id = "your ad application client id" in custom.tfvars and do not run these commands.
+TF_VAR_application_clientid=$(az keyvault secret show --id https://$AZURE_VAULT.vault.azure.net/secrets/${ADO_PROJECT}-${UNIQUE}-application-clientid --query value -otsv)
+echo -e "aad_client_id = \"$TF_VAR_application_clientid\"" >> custom.tfvars
 ```
 
 Execute the following commands to orchestrate a deployment.

infra/templates/osdu-r3-mvp/central_resources/main.tf

@@ -323,7 +323,7 @@ module "ad_application" {
     }
   ]
 
-  enable_bring_your_own_ad_app = var.enable_bring_your_own_ad_app
+  aad_client_id = var.aad_client_id
 }
 
 

infra/templates/osdu-r3-mvp/central_resources/secrets.tf

@@ -155,7 +155,7 @@ resource "azurerm_key_vault_secret" "principal_object_id" {
 // Add Application Information to KV
 resource "azurerm_key_vault_secret" "application_id" {
   name         = "aad-client-id"
-  value        = var.enable_bring_your_own_ad_app ? null : module.ad_application.id
+  value        = var.aad_client_id != "" ? var.aad_client_id : module.ad_application.id
   key_vault_id = module.keyvault.keyvault_id
 }
 

infra/templates/osdu-r3-mvp/central_resources/variables.tf

@@ -40,10 +40,10 @@ variable "feature_flag" {
   }
 }
 
-variable "enable_bring_your_own_ad_app" {
-  description = "Feature flag for BYOA"
-  default     = false
-  type        = bool
+variable "aad_client_id" {
+  description = "Existing Application AppId."
+  type        = string
+  default     = ""
 }
 
 variable "randomization_level" {