From 42d153cfb0a96fe181d6f6acc094e49d201b62fa Mon Sep 17 00:00:00 2001 From: Daniel Scholl Date: Mon, 19 Oct 2020 14:29:50 -0500 Subject: [PATCH 1/9] Added in additional items for Service Pipelines --- README.md | 22 +++++++++- docs/pipeline-setup.md | 90 ++++++++++++++++++++++++++++------------- infra/common_prepare.sh | 5 +++ 3 files changed, 89 insertions(+), 28 deletions(-) diff --git a/README.md b/README.md index 9b1ae772..f38d75e1 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ Azure environment cost ballpark [estimate](https://tinyurl.com/y4e9s7rf). This i 1. Azure Subscription 1. Terraform and Go are locally installed. 1. Requires the use of [direnv](https://direnv.net/). -1. Install the required common tools (kubectl, helm, and terraform). +1. Install the required common tools (kubectl, helm, and terraform). ### Install the required tooling @@ -114,6 +114,26 @@ export UNIQUE=demo ./infra/common_prepare.sh $(az account show --query id -otsv) $UNIQUE ``` +Integration Tests requires 2 Azure AD User Accounts, a tenant user and a guest user to be setup in order to use for integration testing. This activity needs to be performed by someone who has access as an AD User Admin. + +- ad-guest-email (ie: integration.test@email.com) +- ad-guest-oid (OID of the user) +- ad-user-email (ie: integration.test@{tenant}.onmicrosoft.com +- ad-user-oid (OID of the user) + +```bash +USER_EMAIL="" +USER_EMAIL_OID="" +GUEST_EMAIL="" +GUEST_EMAIL_OID="" + +az keyvault secret set --vault-name $COMMON_VAULT --name "ad-user-email" --value $USER_EMAIL +az keyvault secret set --vault-name $COMMON_VAULT --name "ad-user-oid" --value $USER_EMAIL_OID +az keyvault secret set --vault-name $COMMON_VAULT --name "ad-guest-email" --value $GUEST_EMAIL +az keyvault secret set --vault-name $COMMON_VAULT --name "ad-guest-oid" --value $GUEST_EMAIL_OID +``` + + __Local Script Output Resources__ diff --git a/docs/pipeline-setup.md b/docs/pipeline-setup.md index 3fab03ee..b1ccd81b 100644 --- a/docs/pipeline-setup.md +++ b/docs/pipeline-setup.md @@ -1,7 +1,6 @@ -# Configure Continous Deployment for Infrastructure and Services into Environments. +# Configure Continous Deployment for Infrastructure and OSDU Services - -## Setup Mirroring of Gitlab Repositories +## Setup Mirroring for Gitlab Repositories > This typically takes about 10 minutes to complete. @@ -22,8 +21,10 @@ Empty repositories need to be created that will be used by a pipeline to mirror | delivery | https://community.opengroup.org/osdu/platform/system/delivery.git | ```bash -export ADO_ORGANIZATION= # ie: osdu-demo -export ADO_PROJECT= # ie: osdu-mvp +export ADO_ORGANIZATION= +export ADO_PROJECT=osdu-mvp + +az devops configure --defaults organization=https://dev.azure.com/$ADO_ORGANIZATION project=$ADO_PROJECT # Create required ADO Repositories for SERVICE in infra-azure-provisioning partition entitlements-azure legal storage indexer-queue indexer-service search-service delivery; @@ -35,7 +36,7 @@ done __Create Variable Group__ -A variable group will be used to hold the values of the GitLab Location to be mirrored. Additionally a Personal Access Token is necessary to allow for git checkin. +This variable group will be used to hold the values of the GitLab Location to be mirrored. Additionally a Personal Access Token is necessary to allow for git checkin. Variable Group Name: `Mirror Variables` @@ -63,8 +64,6 @@ ACCESS_TOKEN= az pipelines variable-group create \ --name "Mirror Variables" \ --authorize true \ - --organization https://dev.azure.com/${ADO_ORGANIZATION} \ - --project $ADO_PROJECT \ --variables \ INFRA_PROVISIONING_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/infra-azure-provisioning \ PARTITION_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/partition \ @@ -197,13 +196,14 @@ az pipelines create \ --branch master \ --repository-type tfsgit \ --yaml-path /pipeline.yml \ - --organization https://dev.azure.com/${ADO_ORGANIZATION} \ - --project $ADO_PROJECT \ -ojson ``` -## Build Azure Resources + + + +## Deploy Infrastructure > This typically takes about 2 hours to complete. @@ -230,6 +230,8 @@ az devops service-endpoint azurerm create \ __Setup and Configure the ADO Library `Infrastructure Pipeline Variables`__ +This variable group will be used to hold the common values for infrastructure to be built regardless of a specified environment. + | Variable | Value | |----------|-------| | AGENT_POOL | Hosted Ubuntu 1604 | @@ -260,7 +262,7 @@ az pipelines variable-group create \ __Setup and Configure the ADO Library `Infrastructure Pipeline Variables - demo`__ -> You can specify the desired region locations you wish. +This variable group will be used to hold the common values for a specific infrastructure environment to be built. There is an implied naming convention to this Variable group `demo` relates to the environment name. Additionally you can specify and override the region locations here. | Variable | Value | @@ -335,7 +337,7 @@ __Setup 2 Secure Files__ __Execute the pipelines in `osdu-infrastructure`__ -> This should be executed to completion in order +> These pipelines need to be executed to completion in the specific order. 1. `infrastructure-central-resources` @@ -394,43 +396,77 @@ az pipelines create \ ## Deploy OSDU Services -> This typically takes about 3-4 hours to complete. +> This typically takes about 1-2 hours to complete. __Setup and Configure the ADO Library `Azure - OSDU`__ +This variable group will be used to hold the common values for the services to be deployed and relate specifically to settings common to integration tests across all services. Variable values here that are other variables are being pulled from the Library group accessing the common Key Vault. + | Variable | Value | |-----------------------------------------------|-------| | ADMIN_EMAIL | | | AGENT_POOL | `Hosted Ubuntu 1604` | | AZURE_AD_GUEST_EMAIL | `$(ad-guest-email)` | | AZURE_AD_GUEST_OID | `$(ad-guest-oid)` | -| AZURE_AD_OTHER_APP_RESOURCE_ID | `$(osdu-infra--test-app-noaccess-id)` | -| AZURE_AD_OTHER_APP_RESOURCE_OID | `$(osdu-infra--test-noaccess-app-oid)`| +| AZURE_AD_OTHER_APP_RESOURCE_ID | `$(osdu-mvp--application-clientid)` | +| AZURE_AD_OTHER_APP_RESOURCE_OID | `$(osdu-mvp--application-oid)`| | AZURE_AD_USER_EMAIL | `$(ad-user-email)` | | AZURE_AD_USER_OID | `$(ad-user-oid)` | | AZURE_LEGAL_TOPICNAME | `legaltags` | | DEPLOY_ENV | `empty` | -| ENTITLEMENT_URL | `https://$(DNS_HOST)/entitlements/v1/` | +| ENTITLEMENT_URL | `https:///entitlements/v1/` | | EXPIRED_TOKEN | | -| HOST_URL | `https://$(DNS_HOST)/` | -| LEGAL_URL | `https://$(DNS_HOST)/api/legal/v1/` | -| NO_DATA_ACCESS_TESTER | `$(osdu-infra-azg-test-app-noaccess-id)` | -| NO_DATA_ACCESS_TESTER_SERVICEPRINCIPAL_SECRET | `$(osdu-infra-azg-test-app-noaccess-key)` | -| PUBSUB_TOKEN | `empty` | +| HOST_URL | `https:///` | +| LEGAL_URL | `https:///api/legal/v1/` | +| NO_DATA_ACCESS_TESTER | `$(osdu-mvp--noaccess-clientid)` | +| NO_DATA_ACCESS_TESTER_SERVICEPRINCIPAL_SECRET | `$(osdu-mvp--noaccess-secret)` | +| PUBSUB_TOKEN | `az` | | SERVICE_CONNECTION_NAME | | | GOOGLE_CLOUD_PROJECT | `opendes` | +```bash +ADMIN_EMAIL="" # ie: admin@email.com +DNS_HOST="" # ie: osdu.contoso.com +INVALID_TOKEN="" # jwt token + +az pipelines variable-group create \ + --name "Azure - OSDU" \ + --authorize true \ + --variables \ + ADMIN_EMAIL=$ADMIN_EMAIL \ + AGENT_POOL="Hosted Ubuntu 1604" \ + AZURE_AD_GUEST_EMAIL="$(ad-guest-email)" \ + AZURE_AD_GUEST_OID="$(ad-guest-oid)" \ + AZURE_AD_OTHER_APP_RESOURCE_ID=$(osdu-mvp-${UNIQUE}-application-clientid) \ + AZURE_AD_OTHER_APP_RESOURCE_OID=$(osdu-mvp-${UNIQUE}-application-oid) \ + AZURE_AD_USER_EMAIL="$(ad-user-email)" \ + AZURE_AD_USER_OID="$(ad-user-oid)" \ + AZURE_LEGAL_TOPICNAME="legaltags" \ + DEPLOY_ENV="empty" \ + ENTITLEMENT_URL="https://${DNS_HOST}/entitlements/v1/" \ + EXPIRED_TOKEN=$INVALID_TOKEN \ + HOST_URL="https://${DNS_HOST}/" \ + LEGAL_URL="https://${DNS_HOST}/api/legal/v1/" \ + NO_DATA_ACCESS_TESTER="$(osdu-mvp-${UNIQUE}-noaccess-clientid)" \ + NO_DATA_ACCESS_TESTER_SERVICEPRINCIPAL_SECRET="$(osdu-mvp-${UNIQUE}-noaccess-secret))" \ + PUBSUB_TOKEN="az" \ + SERVICE_CONNECTION_NAME=$SERVICE_CONNECTION_NAME \ + GOOGLE_CLOUD_PROJECT="opendes" + -ojson +``` + + __Setup and Configure the ADO Library `Azure - OSDU Secrets`__ -> This Library is linked to the Common Key Vault +This variable group is a linked variable group that links to the Common Key Vault and retrieves the necessary values. -- osdu-infra-{unique}-test-app-id -- osdu-infra-{unique}-test-app-key -- osdu-infra-{unique}-test-app-noaccess-id -- osdu-infra-{unique}-test-app-noaccess-oid +- osdu-mvp-{unique}-application-clientid +- osdu-mvp-{unique}-application-secret +- osdu-infra-{unique}-noaccess-clientid +- osdu-infra-{unique}-noaccess-oid - ad-guest-email - ad-guest-oid - ad-user-email diff --git a/infra/common_prepare.sh b/infra/common_prepare.sh index b6f2982b..04d220e9 100755 --- a/infra/common_prepare.sh +++ b/infra/common_prepare.sh @@ -211,9 +211,14 @@ function CreateADApplication() { --display-name $1 \ --query [].appId -otsv) + APP_OID=$(az ad app list \ + --display-name $1 \ + --query [].objectId -otsv) + tput setaf 2; echo "Adding AD Application to Vault..." ; tput sgr0 AddKeyToVault $2 "${1}-clientid" $APP_ID AddKeyToVault $2 "${1}-secret" $APP_SECRET + AddKeyToVault $2 "${1}-oid" $APP_OID else tput setaf 3; echo "AD Application $1 already exists."; tput sgr0 -- GitLab From 3d2387821674145b2797d411c57376e84c7323bd Mon Sep 17 00:00:00 2001 From: Daniel Scholl Date: Tue, 20 Oct 2020 10:50:36 -0500 Subject: [PATCH 2/9] Updated Documentation --- README.md | 9 +++++ docs/pipeline-setup.md | 74 ++++++++++++++++++++++++++++-------------- 2 files changed, 58 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index f38d75e1..ba2409f2 100644 --- a/README.md +++ b/README.md @@ -133,7 +133,16 @@ az keyvault secret set --vault-name $COMMON_VAULT --name "ad-guest-email" --valu az keyvault secret set --vault-name $COMMON_VAULT --name "ad-guest-oid" --value $GUEST_EMAIL_OID ``` +Istio Configuration setups a Dashboard that requires some admin credentials. Automation Pipelines uses settings out of the common keyvault for applying the values of the Istio Dashboard default credentials. +```bash +ISTIO_USERNAME="" +ISTIO_PASSWORD="" + + +az keyvault secret set --vault-name $COMMON_VAULT --name "istio-username" --value $(echo $ISTIO_USERNAME |base64) +az keyvault secret set --vault-name $COMMON_VAULT --name "istio-password" --value $(echo $ISTIO_PASSWORD |base64) +``` __Local Script Output Resources__ diff --git a/docs/pipeline-setup.md b/docs/pipeline-setup.md index b1ccd81b..7c7fd698 100644 --- a/docs/pipeline-setup.md +++ b/docs/pipeline-setup.md @@ -429,7 +429,8 @@ This variable group will be used to hold the common values for the services to b ```bash ADMIN_EMAIL="" # ie: admin@email.com DNS_HOST="" # ie: osdu.contoso.com -INVALID_TOKEN="" # jwt token +SERVICE_CONNECTION_NAME=osdu-mvp-$UNIQUE +INVALID_TOKEN="" az pipelines variable-group create \ --name "Azure - OSDU" \ @@ -437,48 +438,46 @@ az pipelines variable-group create \ --variables \ ADMIN_EMAIL=$ADMIN_EMAIL \ AGENT_POOL="Hosted Ubuntu 1604" \ - AZURE_AD_GUEST_EMAIL="$(ad-guest-email)" \ - AZURE_AD_GUEST_OID="$(ad-guest-oid)" \ - AZURE_AD_OTHER_APP_RESOURCE_ID=$(osdu-mvp-${UNIQUE}-application-clientid) \ - AZURE_AD_OTHER_APP_RESOURCE_OID=$(osdu-mvp-${UNIQUE}-application-oid) \ - AZURE_AD_USER_EMAIL="$(ad-user-email)" \ - AZURE_AD_USER_OID="$(ad-user-oid)" \ + AZURE_AD_GUEST_EMAIL='$(ad-guest-email)' \ + AZURE_AD_GUEST_OID='$(ad-guest-oid)' \ + AZURE_AD_OTHER_APP_RESOURCE_ID='$(osdu-mvp-'${UNIQUE}'-application-clientid)' \ + AZURE_AD_OTHER_APP_RESOURCE_OID='$(osdu-mvp-'${UNIQUE}'-application-oid)' \ + AZURE_AD_USER_EMAIL='$(ad-user-email)' \ + AZURE_AD_USER_OID='$(ad-user-oid)' \ AZURE_LEGAL_TOPICNAME="legaltags" \ DEPLOY_ENV="empty" \ ENTITLEMENT_URL="https://${DNS_HOST}/entitlements/v1/" \ EXPIRED_TOKEN=$INVALID_TOKEN \ HOST_URL="https://${DNS_HOST}/" \ LEGAL_URL="https://${DNS_HOST}/api/legal/v1/" \ - NO_DATA_ACCESS_TESTER="$(osdu-mvp-${UNIQUE}-noaccess-clientid)" \ - NO_DATA_ACCESS_TESTER_SERVICEPRINCIPAL_SECRET="$(osdu-mvp-${UNIQUE}-noaccess-secret))" \ + NO_DATA_ACCESS_TESTER='$(osdu-mvp-'${UNIQUE}'-noaccess-clientid)' \ + NO_DATA_ACCESS_TESTER_SERVICEPRINCIPAL_SECRET='$(osdu-mvp-'${UNIQUE}'-noaccess-secret))' \ PUBSUB_TOKEN="az" \ SERVICE_CONNECTION_NAME=$SERVICE_CONNECTION_NAME \ - GOOGLE_CLOUD_PROJECT="opendes" + GOOGLE_CLOUD_PROJECT="opendes" \ -ojson ``` - __Setup and Configure the ADO Library `Azure - OSDU Secrets`__ -This variable group is a linked variable group that links to the Common Key Vault and retrieves the necessary values. +This variable group is a linked variable group that links to the Common Key Vault `osducommon` and retrieves secret common settings. -- osdu-mvp-{unique}-application-clientid -- osdu-mvp-{unique}-application-secret -- osdu-infra-{unique}-noaccess-clientid -- osdu-infra-{unique}-noaccess-oid - ad-guest-email - ad-guest-oid - ad-user-email - ad-user-oid - istio-username - istio-password - +- osdu-mvp-{unique}-application-clientid +- osdu-mvp-{unique}-application-secret +- osdu-infra-{unique}-noaccess-clientid +- osdu-infra-{unique}-noaccess-oid __Setup and Configure the ADO Library `Azure Target Env - demo`__ -> This library is subject to change due to pipeline tranformation work not completed. +This variable group will be used to hold the specific environment values necessary for integration testing of the services being deployed. Variable values here that are other variables are being pulled from the Library group accessing the environment Key Vault. | Variable | Value | |----------|-------| @@ -498,9 +497,34 @@ __Setup and Configure the ADO Library `Azure Target Env - demo`__ | STORAGE_ACCOUNT_KEY | `$(opendes-storage-key)` | +```bash +DATA_PARTITION_NAME=opendes +DNS_HOST="" # ie: osdu.contoso.com + +az pipelines variable-group create \ + --name "Azure Target Env - ${UNIQUE}" \ + --authorize true \ + --variables \ + AZURE_AD_APP_RESOURCE_ID='$(aad-client-id)' \ + AZURE_DEPLOY_SUBSCRIPTION='$(subscription-id)' \ + AZURE_LEGAL_SERVICEBUS='$('${DATA_PARTITION_NAME}'-sb-connection)' \ + AZURE_TENANT_ID='$(tenant-id)' \ + AZURE_TESTER_SERVICEPRINCIPAL_SECRET='$(app-dev-sp-password)' \ + CONTAINER_REGISTRY_NAME='$(container_registry)' \ + DNS_HOST="$DNS_HOST" \ + DOMAIN="contoso.com" \ + ELASTIC_ENDPOINT='$('${DATA_PARTITION_NAME}'-elastic-endpoint)' \ + IDENTITY_CLIENT_ID='$(identity_id)' \ + INTEGRATION_TESTER='$(app-dev-sp-username)' \ + MY_TENANT="$DATA_PARTITION_NAME" \ + STORAGE_ACCOUNT='$('${DATA_PARTITION_NAME}'-storage)' \ + STORAGE_ACCOUNT_KEY='$('${DATA_PARTITION_NAME}'-storage-key)' \ + -ojson +``` + __Setup and Configure the ADO Library `Azure Target Env Secrets - demo`__ -> This Library is linked to the Enviroment Key Vault +This variable group is a linked variable group that links to the Environment Key Vault and retrieves secret common settings. - aad-client-id - app-dev-sp-id @@ -511,12 +535,12 @@ __Setup and Configure the ADO Library `Azure Target Env Secrets - demo`__ - base-name-cr - base-name-sr - container-registry -- opendes-cosmos-connection -- opendes-cosmos-endpoint -- opendes-cosmos-primary-key -- opendes-elastic-endpoint -- opendes-elastic-password -- opendes-elastic-username +- {partition-name}-cosmos-connection +- {partition-name}-cosmos-endpoint +- {partition-name}-cosmos-primary-key +- {partition-name}-elastic-endpoint +- {partition-name}-elastic-password +- {partition-name}-elastic-username - osdu-identity-id - subscription-id - tenant-id -- GitLab From 81e517816be2147fb98d6b968cb275eaa995febc Mon Sep 17 00:00:00 2001 From: Daniel Scholl Date: Tue, 20 Oct 2020 11:34:23 -0500 Subject: [PATCH 3/9] Added Service Library scripts --- docs/pipeline-setup.md | 99 +++++++++++++++++++++++++++++++++++------- 1 file changed, 83 insertions(+), 16 deletions(-) diff --git a/docs/pipeline-setup.md b/docs/pipeline-setup.md index 7c7fd698..dca2b9d2 100644 --- a/docs/pipeline-setup.md +++ b/docs/pipeline-setup.md @@ -200,9 +200,6 @@ az pipelines create \ ``` - - - ## Deploy Infrastructure > This typically takes about 2 hours to complete. @@ -222,8 +219,6 @@ az devops service-endpoint azurerm create \ --azure-rm-subscription-id $ARM_SUBSCRIPTION_ID \ --azure-rm-subscription-name $(az account show --subscription $ARM_SUBSCRIPTION_ID --query name -otsv) \ --azure-rm-service-principal-id $ARM_CLIENT_ID \ - --organization https://dev.azure.com/${ADO_ORGANIZATION} \ - --project $ADO_PROJECT \ -ojsonc ``` @@ -246,8 +241,6 @@ This variable group will be used to hold the common values for infrastructure to az pipelines variable-group create \ --name "Infrastructure Pipeline Variables" \ --authorize true \ - --organization https://dev.azure.com/${ADO_ORGANIZATION} \ - --project $ADO_PROJECT \ --variables \ AGENT_POOL="Hosted Ubuntu 1604" \ BUILD_ARTIFACT_NAME="infra-templates" \ @@ -294,8 +287,6 @@ GIT_REPO=git@ssh.dev.azure.com:v3/${ADO_ORGANIZATION}/${ADO_PROJECT}/k8-gitops-m az pipelines variable-group create \ --name "Infrastructure Pipeline Variables - ${ENVIRONMENT}" \ --authorize true \ - --organization https://dev.azure.com/${ADO_ORGANIZATION} \ - --project $ADO_PROJECT \ --variables \ ARM_SUBSCRIPTION_ID="${ARM_SUBSCRIPTION_ID}" \ TF_VAR_aks_agent_vm_count=3 \ @@ -351,8 +342,6 @@ az pipelines create \ --branch master \ --repository-type tfsgit \ --yaml-path /infra/templates/osdu-r3-mvp/pipeline-central-resources.yml \ - --organization https://dev.azure.com/${ADO_ORGANIZATION} \ - --project $ADO_PROJECT \ -ojson ``` @@ -369,8 +358,6 @@ az pipelines create \ --branch master \ --repository-type tfsgit \ --yaml-path /infra/templates/osdu-r3-mvp/pipeline-data-partition.yml \ - --organization https://dev.azure.com/${ADO_ORGANIZATION} \ - --project $ADO_PROJECT \ -ojson ``` @@ -387,8 +374,6 @@ az pipelines create \ --branch master \ --repository-type tfsgit \ --yaml-path /infra/templates/osdu-r3-mvp/pipeline-service-resources.yml \ - --organization https://dev.azure.com/${ADO_ORGANIZATION} \ - --project $ADO_PROJECT \ -ojson ``` @@ -548,6 +533,8 @@ This variable group is a linked variable group that links to the Environment Key __Setup and Configure the ADO Library `Azure Service Release - partition`__ +This variable group is the service specific variables necessary for testing and deploying the `partition` service. + | Variable | Value | |----------|-------| | MAVEN_DEPLOY_POM_FILE_PATH | `drop/provider/partition-azure` | @@ -556,8 +543,22 @@ __Setup and Configure the ADO Library `Azure Service Release - partition`__ | SERVICE_RESOURCE_NAME | `$(AZURE_PARTITION_SERVICE_NAME)` | +```bash +az pipelines variable-group create \ + --name "Azure Service Release - partition" \ + --authorize true \ + --variables \ + MAVEN_DEPLOY_POM_FILE_PATH="drop/provider/partition-azure" \ + MAVEN_INTEGRATION_TEST_OPTIONS='-DINTEGRATION_TESTER=$(INTEGRATION_TESTER) -DPARTITION_BASE_URL=$(HOST_URL) -DMY_TENANT=$(MY_TENANT) -DAZURE_TESTER_SERVICEPRINCIPAL_SECRET=$(AZURE_TESTER_SERVICEPRINCIPAL_SECRET) -DAZURE_AD_TENANT_ID=$(AZURE_TENANT_ID) -DAZURE_AD_APP_RESOURCE_ID=$(AZURE_AD_APP_RESOURCE_ID) -DNO_DATA_ACCESS_TESTER=$(NO_DATA_ACCESS_TESTER) -DNO_DATA_ACCESS_TESTER_SERVICEPRINCIPAL_SECRET=$(NO_DATA_ACCESS_TESTER_SERVICEPRINCIPAL_SECRET) -DAZURE_AD_OTHER_APP_RESOURCE_ID=$(AZURE_AD_OTHER_APP_RESOURCE_ID) -DENVIRONMENT=HOSTED' \ + MAVEN_INTEGRATION_TEST_POM_FILE_PATH="drop/deploy/testing/partition-test-azure/pom.xml" \ + SERVICE_RESOURCE_NAME='$(AZURE_PARTITION_SERVICE_NAME)' \ + -ojson +``` + __Setup and Configure the ADO Library `Azure Service Release - entitlements-azure`__ +This variable group is the service specific variables necessary for testing and deploying the `entitlements-azure` service. + | Variable | Value | |----------|-------| | MAVEN_DEPLOY_POM_FILE_PATH | `drop` | @@ -565,9 +566,22 @@ __Setup and Configure the ADO Library `Azure Service Release - entitlements-azur | MAVEN_INTEGRATION_TEST_POM_FILE_PATH | `drop/deploy/integration-tests` | | SERVICE_RESOURCE_NAME | `$(AZURE_ENTITLEMENTS_SERVICE_NAME)` | +```bash +az pipelines variable-group create \ + --name "Azure Service Release - entitlements-azure" \ + --authorize true \ + --variables \ + MAVEN_DEPLOY_POM_FILE_PATH="drop" \ + MAVEN_INTEGRATION_TEST_OPTIONS='-DENTITLEMENT_URL=$(ENTITLEMENT_URL) -DMY_TENANT=$(MY_TENANT) -DAZURE_AD_TENANT_ID=$(AZURE_TENANT_ID) -DINTEGRATION_TESTER=$(INTEGRATION_TESTER) -DENTITLEMENT_MEMBER_NAME_VALID=$(INTEGRATION_TESTER) -DAZURE_TESTER_SERVICEPRINCIPAL_SECRET=$(AZURE_TESTER_SERVICEPRINCIPAL_SECRET) -DAZURE_AD_APP_RESOURCE_ID=$(AZURE_AD_APP_RESOURCE_ID) -DAZURE_AD_OTHER_APP_RESOURCE_ID=$(AZURE_AD_OTHER_APP_RESOURCE_ID) -DAZURE_AD_OTHER_APP_RESOURCE_OID=$(AZURE_AD_OTHER_APP_RESOURCE_OID) -DDOMAIN=$(DOMAIN) -DEXPIRED_TOKEN=$(EXPIRED_TOKEN) -DENTITLEMENT_GROUP_NAME_VALID=integ.test.data.creator -DENTITLEMENT_MEMBER_NAME_INVALID=InvalidTestAdmin -DAZURE_AD_USER_EMAIL=$(ad-user-email) -DAZURE_AD_USER_OID=$(ad-user-oid) -DAZURE_AD_GUEST_EMAIL=$(ad-guest-email) -DAZURE_AD_GUEST_OID=$(ad-guest-oid)' \ + MAVEN_INTEGRATION_TEST_POM_FILE_PATH="drop/deploy/integration-tests" \ + SERVICE_RESOURCE_NAME='$(AZURE_ENTITLEMENTS_SERVICE_NAME)' \ + -ojson +``` __Setup and Configure the ADO Library `Azure Service Release - legal`__ +This variable group is the service specific variables necessary for testing and deploying the `legal` service. + | Variable | Value | |----------|-------| | MAVEN_DEPLOY_POM_FILE_PATH | `drop/provider/legal-azure` | @@ -575,9 +589,23 @@ __Setup and Configure the ADO Library `Azure Service Release - legal`__ | MAVEN_INTEGRATION_TEST_POM_FILE_PATH | `drop/deploy/testing/legal-test-azure/pom.xml` | | SERVICE_RESOURCE_NAME | `$(AZURE_LEGAL_SERVICE_NAME)` | +```bash +az pipelines variable-group create \ + --name "Azure Service Release - legal" \ + --authorize true \ + --variables \ + MAVEN_DEPLOY_POM_FILE_PATH="drop/provider/legal-azure" \ + MAVEN_INTEGRATION_TEST_OPTIONS='-DHOST_URL=$(LEGAL_URL) -DENTITLEMENT_URL=$(ENTITLEMENT_URL) -DMY_TENANT=$(MY_TENANT) -DAZURE_AD_TENANT_ID=$(AZURE_TENANT_ID) -DINTEGRATION_TESTER=$(INTEGRATION_TESTER) -DAZURE_TESTER_SERVICEPRINCIPAL_SECRET=$(AZURE_TESTER_SERVICEPRINCIPAL_SECRET) -DAZURE_AD_APP_RESOURCE_ID=$(AZURE_AD_APP_RESOURCE_ID) -DAZURE_LEGAL_STORAGE_ACCOUNT=$(STORAGE_ACCOUNT) -DAZURE_LEGAL_STORAGE_KEY=$(STORAGE_ACCOUNT_KEY) -DAZURE_LEGAL_SERVICEBUS=$(AZURE_LEGAL_SERVICEBUS) -DAZURE_LEGAL_TOPICNAME=$(AZURE_LEGAL_TOPICNAME)' \ + MAVEN_INTEGRATION_TEST_POM_FILE_PATH="drop/deploy/testing/legal-test-azure/pom.xml" \ + SERVICE_RESOURCE_NAME='$(AZURE_LEGAL_SERVICE_NAME)' \ + -ojson +``` + __Setup and Configure the ADO Library `Azure Service Release - storage`__ +This variable group is the service specific variables necessary for testing and deploying the `storage` service. + | Variable | Value | |----------|-------| | MAVEN_DEPLOY_POM_FILE_PATH | `drop/provider/storage-azure` | @@ -585,9 +613,23 @@ __Setup and Configure the ADO Library `Azure Service Release - storage`__ | MAVEN_INTEGRATION_TEST_POM_FILE_PATH | `drop/deploy/testing/storage-test-azure` | | SERVICE_RESOURCE_NAME | `$(AZURE_STORAGE_SERVICE_NAME)` | +```bash +az pipelines variable-group create \ + --name "Azure Service Release - storage" \ + --authorize true \ + --variables \ + MAVEN_DEPLOY_POM_FILE_PATH="drop/provider/storage-azure" \ + MAVEN_INTEGRATION_TEST_OPTIONS='-DSTORAGE_URL=$(STORAGE_URL) -DLEGAL_URL=$(LEGAL_URL) -DTENANT_NAME=$(MY_TENANT) -DAZURE_AD_TENANT_ID=$(AZURE_TENANT_ID) -DINTEGRATION_TESTER=$(INTEGRATION_TESTER) -DTESTER_SERVICEPRINCIPAL_SECRET=$(AZURE_TESTER_SERVICEPRINCIPAL_SECRET) -DAZURE_STORAGE_ACCOUNT=$(STORAGE_ACCOUNT) -DAZURE_AD_APP_RESOURCE_ID=$(AZURE_AD_APP_RESOURCE_ID) -DNO_DATA_ACCESS_TESTER=$(NO_DATA_ACCESS_TESTER) -DNO_DATA_ACCESS_TESTER_SERVICEPRINCIPAL_SECRET=$(NO_DATA_ACCESS_TESTER_SERVICEPRINCIPAL_SECRET) -DDOMAIN=$(DOMAIN) -DPUBSUB_TOKEN=$(PUBSUB_TOKEN) -DDEPLOY_ENV=$(DEPLOY_ENV)' \ + MAVEN_INTEGRATION_TEST_POM_FILE_PATH="drop/deploy/testing/storage-test-azure" \ + SERVICE_RESOURCE_NAME='$(AZURE_STORAGE_SERVICE_NAME)' \ + -ojson +``` + __Setup and Configure the ADO Library `Azure Service Release - indexer-service`__ +This variable group is the service specific variables necessary for testing and deploying the `indexer-service` service. + | Variable | Value | |----------|-------| | MAVEN_DEPLOY_POM_FILE_PATH | `drop/provider/indexer-azure` | @@ -595,9 +637,24 @@ __Setup and Configure the ADO Library `Azure Service Release - indexer-service`_ | MAVEN_INTEGRATION_TEST_POM_FILE_PATH | `drop/deploy/testing/indexer-test-azure` | | SERVICE_RESOURCE_NAME | `$(AZURE_INDEXER_SERVICE_NAME)` | +```bash +az pipelines variable-group create \ + --name "Azure Service Release - indexer-service" \ + --authorize true \ + --variables \ + MAVEN_DEPLOY_POM_FILE_PATH="drop/provider/indexer-azure" \ + MAVEN_INTEGRATION_TEST_OPTIONS='-DAZURE_AD_TENANT_ID=$(AZURE_TENANT_ID) -DINTEGRATION_TESTER=$(INTEGRATION_TESTER) -DAZURE_TESTER_SERVICEPRINCIPAL_SECRET=$(AZURE_TESTER_SERVICEPRINCIPAL_SECRET) -DAZURE_AD_APP_RESOURCE_ID=$(AZURE_AD_APP_RESOURCE_ID) -Daad_client_id=$(AZURE_AD_APP_RESOURCE_ID) -DSTORAGE_HOST=$(STORAGE_URL) -DELASTIC_HOST=$(ELASTIC_HOST) -DELASTIC_PORT=$(ELASTIC_PORT) -DELASTIC_USER_NAME=$(ELASTIC_USERNAME) -DELASTIC_PASSWORD=$(ELASTIC_PASSWORD) -DDEFAULT_DATA_PARTITION_ID_TENANT1=$(MY_TENANT) -DDEFAULT_DATA_PARTITION_ID_TENANT2=othertenant2 -DENTITLEMENTS_DOMAIN=contoso.com -DENVIRONMENT=CLOUD -DLEGAL_TAG=opendes-public-usa-dataset-7643990 -DOTHER_RELEVANT_DATA_COUNTRIES=US' \ + MAVEN_INTEGRATION_TEST_POM_FILE_PATH="drop/deploy/testing/indexer-test-azure" \ + SERVICE_RESOURCE_NAME='$(AZURE_INDEXER_SERVICE_NAME)' \ + -ojson +``` + + __Setup and Configure the ADO Library `Azure Service Release - search-service`__ +This variable group is the service specific variables necessary for testing and deploying the `indexer-service` service. + | Variable | Value | |----------|-------| | MAVEN_DEPLOY_POM_FILE_PATH | `drop/provider/search-azure` | @@ -605,7 +662,17 @@ __Setup and Configure the ADO Library `Azure Service Release - search-service`__ | MAVEN_INTEGRATION_TEST_POM_FILE_PATH | `drop/deploy/testing/integration-tests/search-test-azure` | | SERVICE_RESOURCE_NAME | `$(AZURE_SEARCH_SERVICE_NAME)` | - +```bash +az pipelines variable-group create \ + --name "Azure Service Release - search-service" \ + --authorize true \ + --variables \ + MAVEN_DEPLOY_POM_FILE_PATH="drop/provider/search-azure" \ + MAVEN_INTEGRATION_TEST_OPTIONS='-DSEARCH_HOST=$(SEARCH_URL) -DAZURE_AD_TENANT_ID=$(AZURE_TENANT_ID) -DINTEGRATION_TESTER=$(INTEGRATION_TESTER) -DAZURE_TESTER_SERVICEPRINCIPAL_SECRET=$(AZURE_TESTER_SERVICEPRINCIPAL_SECRET) -DAZURE_AD_APP_RESOURCE_ID=$(AZURE_AD_APP_RESOURCE_ID) -DINDEXER_HOST=$() -DSTORAGE_HOST=$() -DELASTIC_HOST=$(ELASTIC_HOST) -DELASTIC_PORT=$(ELASTIC_PORT) -DELASTIC_USER_NAME=$(ELASTIC_USERNAME) -DELASTIC_PASSWORD=$(ELASTIC_PASSWORD) -DDEFAULT_DATA_PARTITION_ID_TENANT1=$(MY_TENANT) -DDEFAULT_DATA_PARTITION_ID_TENANT2=othertenant2 -DENTITLEMENTS_DOMAIN=$(DOMAIN)' \ + MAVEN_INTEGRATION_TEST_POM_FILE_PATH="drop/deploy/testing/integration-tests/search-test-azure" \ + SERVICE_RESOURCE_NAME='$(AZURE_SEARCH_SERVICE_NAME)' \ + -ojson +``` __Load Storage Container Integration Test Data__ -- GitLab From 526cc4d3f739df6fa77d1e29d12f1a88440f01fb Mon Sep 17 00:00:00 2001 From: Daniel Scholl Date: Tue, 20 Oct 2020 15:47:32 -0500 Subject: [PATCH 4/9] Updated Documentation and added in test data --- README.md | 32 +- docs/code-mirroring.md | 199 ++++++++ docs/infra-automation.md | 176 +++++++ ...ipeline-setup.md => service-automation.md} | 438 +----------------- tools/test_data/Legal_COO.json | 8 + tools/test_data/README.md | 78 ++++ tools/test_data/legal_tag_1.json | 21 + tools/test_data/legal_tag_2.json | 21 + tools/test_data/legal_tag_3.json | 21 + tools/test_data/storage_schema_1.json | 68 +++ tools/test_data/storage_schema_10.json | 68 +++ tools/test_data/storage_schema_11.json | 68 +++ tools/test_data/storage_schema_2.json | 68 +++ tools/test_data/storage_schema_3.json | 68 +++ tools/test_data/storage_schema_4.json | 68 +++ tools/test_data/storage_schema_5.json | 68 +++ tools/test_data/storage_schema_6.json | 68 +++ tools/test_data/storage_schema_7.json | 68 +++ tools/test_data/storage_schema_8.json | 68 +++ tools/test_data/storage_schema_9.json | 68 +++ tools/test_data/tenant_info_1.json | 24 + tools/test_data/tenant_info_2.json | 23 + tools/test_data/upload-data.py | 135 ++++++ tools/test_data/user_info_1.json | 49 ++ tools/test_data/user_info_2.json | 20 + 25 files changed, 1559 insertions(+), 434 deletions(-) create mode 100644 docs/code-mirroring.md create mode 100644 docs/infra-automation.md rename docs/{pipeline-setup.md => service-automation.md} (55%) create mode 100644 tools/test_data/Legal_COO.json create mode 100644 tools/test_data/README.md create mode 100644 tools/test_data/legal_tag_1.json create mode 100644 tools/test_data/legal_tag_2.json create mode 100644 tools/test_data/legal_tag_3.json create mode 100644 tools/test_data/storage_schema_1.json create mode 100644 tools/test_data/storage_schema_10.json create mode 100644 tools/test_data/storage_schema_11.json create mode 100644 tools/test_data/storage_schema_2.json create mode 100644 tools/test_data/storage_schema_3.json create mode 100644 tools/test_data/storage_schema_4.json create mode 100644 tools/test_data/storage_schema_5.json create mode 100644 tools/test_data/storage_schema_6.json create mode 100644 tools/test_data/storage_schema_7.json create mode 100644 tools/test_data/storage_schema_8.json create mode 100644 tools/test_data/storage_schema_9.json create mode 100644 tools/test_data/tenant_info_1.json create mode 100644 tools/test_data/tenant_info_2.json create mode 100755 tools/test_data/upload-data.py create mode 100644 tools/test_data/user_info_1.json create mode 100644 tools/test_data/user_info_2.json diff --git a/README.md b/README.md index ba2409f2..50478b29 100644 --- a/README.md +++ b/README.md @@ -222,27 +222,41 @@ az keyvault secret show \ ``` -## Automated Pipeline Installation +## Install OSDU -> This typically takes about 3 hours to complete. +There are 2 methods that can be chosen to perform installation at this point in time. + +1. Manual Installation -- Typically used when the desire is to manually make modifications to the environment and have full control all updates and deployments. + +2. Pipeline Installation -- Typically used when the need is to only access the Data Platform but allow for automatic upgrades of infrastructure and services. + + +__Manual Installation__ + +> This typically takes about 2 hours to complete. -1. Configure the Pipelines following directions [here](./docs/pipeline-setup.md). +1. Install the Infrastructure following directions [here](./infra/templates/osdu-r3-mvp). 2. Manually configure your DNS_HOST to the IP Address of the environment IP Address. -3. Deploy the application helm charts following the directions [here](). +3. Upload the Integration Test Data following directions [here](./tools/test_data). +3. Deploy the application helm charts following the directions [here](./charts). -## Manual Installation +## Automated Pipeline Installation -> This typically takes about 2 hours to complete. +> This typically takes about 3 hours to complete. -1. Install the Infrastructure following directions [here](./infra/templates/osdu-r3-mvp/README.md). +1. Setup Code Mirroring following directions [here](./docs/code-mirroring.md). -2. Manually configure your DNS_HOST to the IP Address of the environment IP Address. +2. Setup Infrastructure Automation following directions [here](./docs/infra-automation.md). + +3. Manually configure your DNS_HOST to the IP Address of the environment IP Address. + +4. Upload the Integration Test Data following directions [here](./tools/test_data). -3. Deploy the application helm charts following the directions [here](./charts/README.md). +5. Setup Service Automation following directions [here](./docs/service-automation.md). diff --git a/docs/code-mirroring.md b/docs/code-mirroring.md new file mode 100644 index 00000000..4c65083c --- /dev/null +++ b/docs/code-mirroring.md @@ -0,0 +1,199 @@ +# Setup Mirroring for Gitlab Repositories + +> This typically takes about 10 minutes to complete. + +__Create Empty Repositories__ + +Empty repositories need to be created that will be used by a pipeline to mirror gitlab repositories into. + +| Repository Name | Gitlab Location +|---------------------------|---------------------------| +| infra-azure-provisioning | https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning.git | +| partition | https://community.opengroup.org/osdu/platform/system/partition.git | +| entitlements-azure | https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements-azure.git | +| legal | https://community.opengroup.org/osdu/platform/security-and-compliance/legal.git | +| indexer-queue | https://community.opengroup.org/osdu/platform/system/indexer-queue.git | +| storage | https://community.opengroup.org/osdu/platform/system/storage.git | +| indexer-service | https://community.opengroup.org/osdu/platform/system/indexer-service.git | +| search-service | https://community.opengroup.org/osdu/platform/system/search-service.git | +| delivery | https://community.opengroup.org/osdu/platform/system/delivery.git | + +```bash +export ADO_ORGANIZATION= +export ADO_PROJECT=osdu-mvp + +az devops configure --defaults organization=https://dev.azure.com/$ADO_ORGANIZATION project=$ADO_PROJECT + +# Create required ADO Repositories +for SERVICE in infra-azure-provisioning partition entitlements-azure legal storage indexer-queue indexer-service search-service delivery; +do + az repos create --name $SERVICE --organization https://dev.azure.com/${ADO_ORGANIZATION} --project $ADO_PROJECT -ojson +done +``` + + +__Create Variable Group__ + +This variable group will be used to hold the values of the GitLab Location to be mirrored. Additionally a Personal Access Token is necessary to allow for git checkin. + +Variable Group Name: `Mirror Variables` + +| Variable | Value | +|----------|-------| +| OSDU_INFRASTRUCTURE | https://dev.azure.com/osdu-demo/osdu/_git/osdu-infrastructure | +| INFRA_PROVISIONING_REPO | https://dev.azure.com/osdu-demo/osdu/_git/infra-azure-provisioning | +| PARTITION_REPO | https://dev.azure.com/osdu-demo/osdu/_git/partition | +| ENTITLEMENTS_REPO | https://dev.azure.com/osdu-demo/osdu/_git/entitlements-azure | +| LEGAL_REPO | https://dev.azure.com/osdu-demo/osdu/_git/legal | +| STORAGE_REPO | https://dev.azure.com/osdu-demo/osdu/_git/storage | +| INDEXER_QUEUE_REPO | https://dev.azure.com/osdu-demo/osdu/_git/indexer-queue | +| INDEXER_REPO | https://dev.azure.com/osdu-demo/osdu/_git/indexer-service | +| SEARCH_REPO | https://dev.azure.com/osdu-demo/osdu/_git/search-service | +| DELIVERY_REPO | https://dev.azure.com/osdu-demo/osdu/_git/delivery | +| ACCESS_TOKEN | | + + +Manually create a Personal Access Token following the [documentation](https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&tabs=preview-page) and add a Variable called `ACCESS_TOKEN` with the value being the PAT created. + + +```bash +ACCESS_TOKEN= + +az pipelines variable-group create \ + --name "Mirror Variables" \ + --authorize true \ + --variables \ + INFRA_PROVISIONING_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/infra-azure-provisioning \ + PARTITION_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/partition \ + ENTITLEMENTS_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/entitlements-azure \ + LEGAL_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/legal \ + STORAGE_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/storage \ + INDEXER_QUEUE_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/indexer-queue \ + INDEXER_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/indexer-service \ + SEARCH_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/search-service \ + DELIVERY_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/delivery \ + ACCESS_TOKEN=$ACCESS_TOKEN \ + -ojson +``` + + +__Create Mirror Pipeline__ + +Clone the Project Repository `osdu-mvp`, and add the pipeline. + + +```bash +GIT_SSH_COMMAND="ssh -i ${TF_VAR_gitops_ssh_key_file}" \ + git clone git@ssh.dev.azure.com:v3/${ADO_ORGANIZATION}/${ADO_PROJECT}/${ADO_PROJECT} + +cat > ${ADO_PROJECT}/pipeline.yml << 'EOF' +# Copyright © Microsoft Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +### UNCOMMENT IF YOU WANT A SCHEDULED PULL #### + +# schedules: +# - cron: "*/10 * * * *" +# displayName: Hourly Pull Schedule +# branches: +# include: +# - master +# always: true + +variables: + - group: 'Mirror Variables' + +jobs: + - job: mirror_sync + displayName: 'Pull Repositories' + steps: + + - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1 + displayName: 'infra-azure-provisioning' + inputs: + sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning.git' + destinationGitRepositoryUri: '$(INFRA_PROVISIONING_REPO)' + destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN) + + - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1 + displayName: 'partition' + inputs: + sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/system/partition.git' + destinationGitRepositoryUri: '$(PARTITION_REPO)' + destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN) + + - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1 + displayName: 'entitlements-azure' + inputs: + sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements-azure.git' + destinationGitRepositoryUri: '$(ENTITLEMENTS_REPO)' + destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN) + + - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1 + displayName: 'legal' + inputs: + sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/security-and-compliance/legal.git' + destinationGitRepositoryUri: '$(LEGAL_REPO)' + destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN) + + - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1 + displayName: 'indexer-queue' + inputs: + sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/system/indexer-queue.git' + destinationGitRepositoryUri: '$(INDEXER_QUEUE_REPO)' + destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN) + + - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1 + displayName: 'storage' + inputs: + sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/system/storage.git' + destinationGitRepositoryUri: '$(STORAGE_REPO)' + destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN) + + - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1 + displayName: 'indexer-service' + inputs: + sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/system/indexer-service.git' + destinationGitRepositoryUri: '$(INDEXER_REPO)' + destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN) + + - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1 + displayName: 'search-service' + inputs: + sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/system/search-service.git' + destinationGitRepositoryUri: '$(SEARCH_REPO)' + destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN) + + - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1 + displayName: 'delivery' + inputs: + sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/system/delivery.git' + destinationGitRepositoryUri: '$(DELIVERY_REPO)' + destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN) +EOF + +(cd ${ADO_PROJECT} && git add -A && git commit -m "pipeline" && git push) +rm -rf ${ADO_PROJECT} + +# Create and Execute the Pipeline +az pipelines create \ + --name 'gitlab-sync' \ + --repository $ADO_PROJECT \ + --branch master \ + --repository-type tfsgit \ + --yaml-path /pipeline.yml \ + -ojson +``` + diff --git a/docs/infra-automation.md b/docs/infra-automation.md new file mode 100644 index 00000000..6644db6f --- /dev/null +++ b/docs/infra-automation.md @@ -0,0 +1,176 @@ +# Deploy Infrastructure + +> This typically takes about 2 hours to complete. + +__Configure Azure DevOps Service Connection__ + +- Configure an [ARM Resources Service Connection](https://docs.microsoft.com/en-us/azure/devops/pipelines/library/connect-to-azure?view=azure-devops) for the desired subscription. + - Scope should be to the desired Subscription but do not apply scope to a Resource Group + +```bash +SERVICE_CONNECTION_NAME=osdu-mvp-$UNIQUE +export AZURE_DEVOPS_EXT_AZURE_RM_SERVICE_PRINCIPAL_KEY=$ARM_CLIENT_SECRET + +az devops service-endpoint azurerm create \ + --name $SERVICE_CONNECTION_NAME \ + --azure-rm-tenant-id $ARM_TENANT_ID \ + --azure-rm-subscription-id $ARM_SUBSCRIPTION_ID \ + --azure-rm-subscription-name $(az account show --subscription $ARM_SUBSCRIPTION_ID --query name -otsv) \ + --azure-rm-service-principal-id $ARM_CLIENT_ID \ + -ojsonc +``` + + +__Setup and Configure the ADO Library `Infrastructure Pipeline Variables`__ + +This variable group will be used to hold the common values for infrastructure to be built regardless of a specified environment. + + | Variable | Value | + |----------|-------| + | AGENT_POOL | Hosted Ubuntu 1604 | + | BUILD_ARTIFACT_NAME | infra-templates | + | SERVICE_CONNECTION_NAME | | + | TF_VAR_elasticsearch_secrets_keyvault_name | osducommon-kv | + | TF_VAR_elasticsearch_secrets_keyvault_resource_group | osdu-common- | + | TF_VAR_remote_state_account | osducommon | + | TF_VAR_remote_state_container | remote-state-container | + +```bash +az pipelines variable-group create \ + --name "Infrastructure Pipeline Variables" \ + --authorize true \ + --variables \ + AGENT_POOL="Hosted Ubuntu 1604" \ + BUILD_ARTIFACT_NAME="infra-templates" \ + TF_VAR_elasticsearch_secrets_keyvault_name=$COMMON_VAULT \ + TF_VAR_elasticsearch_secrets_keyvault_resource_group=osdu-common-${UNIQUE} \ + TF_VAR_remote_state_account=$TF_VAR_remote_state_account \ + TF_VAR_remote_state_container="remote-state-container" \ + SERVICE_CONNECTION_NAME=$SERVICE_CONNECTION_NAME \ + -ojson +``` + + +__Setup and Configure the ADO Library `Infrastructure Pipeline Variables - demo`__ + +This variable group will be used to hold the common values for a specific infrastructure environment to be built. There is an implied naming convention to this Variable group `demo` relates to the environment name. Additionally you can specify and override the region locations here. + + + | Variable | Value | + |----------|-------| + | ARM_SUBSCRIPTION_ID | | + | TF_VAR_aks_agent_vm_count | 3 | + | TF_VAR_central_resources_workspace_name | cr-demo | + | TF_VAR_cosmosdb_replica_location | eastus2 | + | TF_VAR_data_partition_name | opendes | + | TF_VAR_data_resources_workspace_name | dr-demo | + | TF_VAR_elasticsearch_version | | + | TF_VAR_gitops_branch | master | + | TF_VAR_gitops_path | providers/azure/hld-registry | + | TF_VAR_gitops_ssh_url | git@ | + | TF_VAR_principal_appId | | + | TF_VAR_principal_name | | + | TF_VAR_principal_objectId | | + | TF_VAR_principal_password | | + | TF_VAR_resource_group_location | centralus | + +```bash +ENVIRONMENT="demo" +REGION="centralus" +REGION_PAIR="eastus2" +PARTITION_NAME="opendes" +ELASTIC_VERSION="6.8.12" +GIT_REPO=git@ssh.dev.azure.com:v3/${ADO_ORGANIZATION}/${ADO_PROJECT}/k8-gitops-manifests + +az pipelines variable-group create \ + --name "Infrastructure Pipeline Variables - ${ENVIRONMENT}" \ + --authorize true \ + --variables \ + ARM_SUBSCRIPTION_ID="${ARM_SUBSCRIPTION_ID}" \ + TF_VAR_aks_agent_vm_count=3 \ + TF_VAR_central_resources_workspace_name="cr-${ENVIRONMENT}" \ + TF_VAR_cosmosdb_replica_location="${REGION_PAIR}" \ + TF_VAR_data_partition_name="${PARTITION_NAME}" \ + TF_VAR_data_resources_workspace_name="dr-${ENVIRONMENT}" \ + TF_VAR_elasticsearch_version="${ELASTIC_VERSION}" \ + TF_VAR_gitops_branch="master" \ + TF_VAR_gitops_path="providers/azure/hld-registry" \ + TF_VAR_gitops_ssh_url="${GIT_REPO}" \ + TF_VAR_principal_appId="${TF_VAR_principal_appId}" \ + TF_VAR_principal_name="${TF_VAR_principal_name}" \ + TF_VAR_principal_objectId="${TF_VAR_principal_objectId}" \ + TF_VAR_principal_password="${TF_VAR_principal_password}" \ + TF_VAR_resource_group_location="${REGION}" \ + -ojson +``` + +__Setup and Configure the ADO Library `Infrastructure Pipeline Secrets - demo`__ +> This should be linked Secrets from Azure Key Vault `osducommon` + + | Variable | Value | + |----------|-------| + | elastic-endpoint-dp1-demo | `*********` | + | elastic-username-dp1-demo | `*********` | + | elastic-password-dp1-demo | `*********` | + + +__Setup 2 Secure Files__ + +[Upload the 2 Secure files](https://docs.microsoft.com/en-us/azure/devops/pipelines/library/secure-files?view=azure-devops). + + + - ~/.ssh/osdu_$UNIQUE/azure-aks-gitops-ssh-key + - ~/.ssh/osdu_$UNIQUE/azure-aks-node-ssh-key.pub + + + +__Execute the pipelines in `osdu-infrastructure`__ + +> These pipelines need to be executed to completion in the specific order. + +1. `infrastructure-central-resources` + + > For the first run of the pipeline approvals will need to be made for the 2 secure files and the Service Connection. + +```bash +# Create and Deploy the Pipeline +az pipelines create \ + --name 'infrastructure-central-resources' \ + --repository infra-azure-provisioning \ + --branch master \ + --repository-type tfsgit \ + --yaml-path /infra/templates/osdu-r3-mvp/pipeline-central-resources.yml \ + -ojson +``` + + +2. `infrastructure-data-partition` + + > For the first run of the pipeline approvals will need to be made for the 2 secure files and the Service Connection. + +```bash +# Create and Deploy the Pipeline +az pipelines create \ + --name 'infrastructure-data-partition' \ + --repository infra-azure-provisioning \ + --branch master \ + --repository-type tfsgit \ + --yaml-path /infra/templates/osdu-r3-mvp/pipeline-data-partition.yml \ + -ojson +``` + + +3. `azure-pipeline-service.yml` + + > For the first run of the pipeline approvals will need to be made for the 2 secure files and the Service Connection. + +```bash +# Create and Deploy the Pipeline +az pipelines create \ + --name 'infrastructure-service-resources' \ + --repository infra-azure-provisioning \ + --branch master \ + --repository-type tfsgit \ + --yaml-path /infra/templates/osdu-r3-mvp/pipeline-service-resources.yml \ + -ojson +``` diff --git a/docs/pipeline-setup.md b/docs/service-automation.md similarity index 55% rename from docs/pipeline-setup.md rename to docs/service-automation.md index dca2b9d2..be9ed5c2 100644 --- a/docs/pipeline-setup.md +++ b/docs/service-automation.md @@ -1,389 +1,7 @@ -# Configure Continous Deployment for Infrastructure and OSDU Services - -## Setup Mirroring for Gitlab Repositories +## Create OSDU Service Libraries > This typically takes about 10 minutes to complete. -__Create Empty Repositories__ - -Empty repositories need to be created that will be used by a pipeline to mirror gitlab repositories into. - -| Repository Name | Gitlab Location -|---------------------------|---------------------------| -| infra-azure-provisioning | https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning.git | -| partition | https://community.opengroup.org/osdu/platform/system/partition.git | -| entitlements-azure | https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements-azure.git | -| legal | https://community.opengroup.org/osdu/platform/security-and-compliance/legal.git | -| indexer-queue | https://community.opengroup.org/osdu/platform/system/indexer-queue.git | -| storage | https://community.opengroup.org/osdu/platform/system/storage.git | -| indexer-service | https://community.opengroup.org/osdu/platform/system/indexer-service.git | -| search-service | https://community.opengroup.org/osdu/platform/system/search-service.git | -| delivery | https://community.opengroup.org/osdu/platform/system/delivery.git | - -```bash -export ADO_ORGANIZATION= -export ADO_PROJECT=osdu-mvp - -az devops configure --defaults organization=https://dev.azure.com/$ADO_ORGANIZATION project=$ADO_PROJECT - -# Create required ADO Repositories -for SERVICE in infra-azure-provisioning partition entitlements-azure legal storage indexer-queue indexer-service search-service delivery; -do - az repos create --name $SERVICE --organization https://dev.azure.com/${ADO_ORGANIZATION} --project $ADO_PROJECT -ojson -done -``` - - -__Create Variable Group__ - -This variable group will be used to hold the values of the GitLab Location to be mirrored. Additionally a Personal Access Token is necessary to allow for git checkin. - -Variable Group Name: `Mirror Variables` - -| Variable | Value | -|----------|-------| -| OSDU_INFRASTRUCTURE | https://dev.azure.com/osdu-demo/osdu/_git/osdu-infrastructure | -| INFRA_PROVISIONING_REPO | https://dev.azure.com/osdu-demo/osdu/_git/infra-azure-provisioning | -| PARTITION_REPO | https://dev.azure.com/osdu-demo/osdu/_git/partition | -| ENTITLEMENTS_REPO | https://dev.azure.com/osdu-demo/osdu/_git/entitlements-azure | -| LEGAL_REPO | https://dev.azure.com/osdu-demo/osdu/_git/legal | -| STORAGE_REPO | https://dev.azure.com/osdu-demo/osdu/_git/storage | -| INDEXER_QUEUE_REPO | https://dev.azure.com/osdu-demo/osdu/_git/indexer-queue | -| INDEXER_REPO | https://dev.azure.com/osdu-demo/osdu/_git/indexer-service | -| SEARCH_REPO | https://dev.azure.com/osdu-demo/osdu/_git/search-service | -| DELIVERY_REPO | https://dev.azure.com/osdu-demo/osdu/_git/delivery | -| ACCESS_TOKEN | | - - -Manually create a Personal Access Token following the [documentation](https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&tabs=preview-page) and add a Variable called `ACCESS_TOKEN` with the value being the PAT created. - - -```bash -ACCESS_TOKEN= - -az pipelines variable-group create \ - --name "Mirror Variables" \ - --authorize true \ - --variables \ - INFRA_PROVISIONING_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/infra-azure-provisioning \ - PARTITION_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/partition \ - ENTITLEMENTS_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/entitlements-azure \ - LEGAL_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/legal \ - STORAGE_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/storage \ - INDEXER_QUEUE_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/indexer-queue \ - INDEXER_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/indexer-service \ - SEARCH_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/search-service \ - DELIVERY_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/delivery \ - ACCESS_TOKEN=$ACCESS_TOKEN \ - -ojson -``` - - -__Create Mirror Pipeline__ - -Clone the Project Repository `osdu-mvp`, and add the pipeline. - - -```bash -GIT_SSH_COMMAND="ssh -i ${TF_VAR_gitops_ssh_key_file}" \ - git clone git@ssh.dev.azure.com:v3/${ADO_ORGANIZATION}/${ADO_PROJECT}/${ADO_PROJECT} - -cat > ${ADO_PROJECT}/pipeline.yml << 'EOF' -# Copyright © Microsoft Corporation -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - - -### UNCOMMENT IF YOU WANT A SCHEDULED PULL #### - -# schedules: -# - cron: "*/10 * * * *" -# displayName: Hourly Pull Schedule -# branches: -# include: -# - master -# always: true - -variables: - - group: 'Mirror Variables' - -jobs: - - job: mirror_sync - displayName: 'Pull Repositories' - steps: - - - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1 - displayName: 'infra-azure-provisioning' - inputs: - sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning.git' - destinationGitRepositoryUri: '$(INFRA_PROVISIONING_REPO)' - destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN) - - - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1 - displayName: 'partition' - inputs: - sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/system/partition.git' - destinationGitRepositoryUri: '$(PARTITION_REPO)' - destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN) - - - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1 - displayName: 'entitlements-azure' - inputs: - sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements-azure.git' - destinationGitRepositoryUri: '$(ENTITLEMENTS_REPO)' - destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN) - - - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1 - displayName: 'legal' - inputs: - sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/security-and-compliance/legal.git' - destinationGitRepositoryUri: '$(LEGAL_REPO)' - destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN) - - - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1 - displayName: 'indexer-queue' - inputs: - sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/system/indexer-queue.git' - destinationGitRepositoryUri: '$(INDEXER_QUEUE_REPO)' - destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN) - - - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1 - displayName: 'storage' - inputs: - sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/system/storage.git' - destinationGitRepositoryUri: '$(STORAGE_REPO)' - destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN) - - - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1 - displayName: 'indexer-service' - inputs: - sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/system/indexer-service.git' - destinationGitRepositoryUri: '$(INDEXER_REPO)' - destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN) - - - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1 - displayName: 'search-service' - inputs: - sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/system/search-service.git' - destinationGitRepositoryUri: '$(SEARCH_REPO)' - destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN) - - - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1 - displayName: 'delivery' - inputs: - sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/system/delivery.git' - destinationGitRepositoryUri: '$(DELIVERY_REPO)' - destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN) -EOF - -(cd ${ADO_PROJECT} && git add -A && git commit -m "pipeline" && git push) -rm -rf ${ADO_PROJECT} - -# Create and Execute the Pipeline -az pipelines create \ - --name 'gitlab-sync' \ - --repository $ADO_PROJECT \ - --branch master \ - --repository-type tfsgit \ - --yaml-path /pipeline.yml \ - -ojson -``` - - -## Deploy Infrastructure - -> This typically takes about 2 hours to complete. - -__Configure Azure DevOps Service Connection__ - -- Configure an [ARM Resources Service Connection](https://docs.microsoft.com/en-us/azure/devops/pipelines/library/connect-to-azure?view=azure-devops) for the desired subscription. - - Scope should be to the desired Subscription but do not apply scope to a Resource Group - -```bash -SERVICE_CONNECTION_NAME=osdu-mvp-$UNIQUE -export AZURE_DEVOPS_EXT_AZURE_RM_SERVICE_PRINCIPAL_KEY=$ARM_CLIENT_SECRET - -az devops service-endpoint azurerm create \ - --name $SERVICE_CONNECTION_NAME \ - --azure-rm-tenant-id $ARM_TENANT_ID \ - --azure-rm-subscription-id $ARM_SUBSCRIPTION_ID \ - --azure-rm-subscription-name $(az account show --subscription $ARM_SUBSCRIPTION_ID --query name -otsv) \ - --azure-rm-service-principal-id $ARM_CLIENT_ID \ - -ojsonc -``` - - -__Setup and Configure the ADO Library `Infrastructure Pipeline Variables`__ - -This variable group will be used to hold the common values for infrastructure to be built regardless of a specified environment. - - | Variable | Value | - |----------|-------| - | AGENT_POOL | Hosted Ubuntu 1604 | - | BUILD_ARTIFACT_NAME | infra-templates | - | SERVICE_CONNECTION_NAME | | - | TF_VAR_elasticsearch_secrets_keyvault_name | osducommon-kv | - | TF_VAR_elasticsearch_secrets_keyvault_resource_group | osdu-common- | - | TF_VAR_remote_state_account | osducommon | - | TF_VAR_remote_state_container | remote-state-container | - -```bash -az pipelines variable-group create \ - --name "Infrastructure Pipeline Variables" \ - --authorize true \ - --variables \ - AGENT_POOL="Hosted Ubuntu 1604" \ - BUILD_ARTIFACT_NAME="infra-templates" \ - TF_VAR_elasticsearch_secrets_keyvault_name=$COMMON_VAULT \ - TF_VAR_elasticsearch_secrets_keyvault_resource_group=osdu-common-${UNIQUE} \ - TF_VAR_remote_state_account=$TF_VAR_remote_state_account \ - TF_VAR_remote_state_container="remote-state-container" \ - SERVICE_CONNECTION_NAME=$SERVICE_CONNECTION_NAME \ - -ojson -``` - - -__Setup and Configure the ADO Library `Infrastructure Pipeline Variables - demo`__ - -This variable group will be used to hold the common values for a specific infrastructure environment to be built. There is an implied naming convention to this Variable group `demo` relates to the environment name. Additionally you can specify and override the region locations here. - - - | Variable | Value | - |----------|-------| - | ARM_SUBSCRIPTION_ID | | - | TF_VAR_aks_agent_vm_count | 3 | - | TF_VAR_central_resources_workspace_name | cr-demo | - | TF_VAR_cosmosdb_replica_location | eastus2 | - | TF_VAR_data_partition_name | opendes | - | TF_VAR_data_resources_workspace_name | dr-demo | - | TF_VAR_elasticsearch_version | | - | TF_VAR_gitops_branch | master | - | TF_VAR_gitops_path | providers/azure/hld-registry | - | TF_VAR_gitops_ssh_url | git@ | - | TF_VAR_principal_appId | | - | TF_VAR_principal_name | | - | TF_VAR_principal_objectId | | - | TF_VAR_principal_password | | - | TF_VAR_resource_group_location | centralus | - -```bash -ENVIRONMENT="demo" -REGION="centralus" -REGION_PAIR="eastus2" -PARTITION_NAME="opendes" -ELASTIC_VERSION="6.8.12" -GIT_REPO=git@ssh.dev.azure.com:v3/${ADO_ORGANIZATION}/${ADO_PROJECT}/k8-gitops-manifests - -az pipelines variable-group create \ - --name "Infrastructure Pipeline Variables - ${ENVIRONMENT}" \ - --authorize true \ - --variables \ - ARM_SUBSCRIPTION_ID="${ARM_SUBSCRIPTION_ID}" \ - TF_VAR_aks_agent_vm_count=3 \ - TF_VAR_central_resources_workspace_name="cr-${ENVIRONMENT}" \ - TF_VAR_cosmosdb_replica_location="${REGION_PAIR}" \ - TF_VAR_data_partition_name="${PARTITION_NAME}" \ - TF_VAR_data_resources_workspace_name="dr-${ENVIRONMENT}" \ - TF_VAR_elasticsearch_version="${ELASTIC_VERSION}" \ - TF_VAR_gitops_branch="master" \ - TF_VAR_gitops_path="providers/azure/hld-registry" \ - TF_VAR_gitops_ssh_url="${GIT_REPO}" \ - TF_VAR_principal_appId="${TF_VAR_principal_appId}" \ - TF_VAR_principal_name="${TF_VAR_principal_name}" \ - TF_VAR_principal_objectId="${TF_VAR_principal_objectId}" \ - TF_VAR_principal_password="${TF_VAR_principal_password}" \ - TF_VAR_resource_group_location="${REGION}" \ - -ojson -``` - -__Setup and Configure the ADO Library `Infrastructure Pipeline Secrets - demo`__ -> This should be linked Secrets from Azure Key Vault `osducommon` - - | Variable | Value | - |----------|-------| - | elastic-endpoint-dp1-demo | `*********` | - | elastic-username-dp1-demo | `*********` | - | elastic-password-dp1-demo | `*********` | - - -__Setup 2 Secure Files__ - -[Upload the 2 Secure files](https://docs.microsoft.com/en-us/azure/devops/pipelines/library/secure-files?view=azure-devops). - - - - ~/.ssh/osdu_$UNIQUE/azure-aks-gitops-ssh-key - - ~/.ssh/osdu_$UNIQUE/azure-aks-node-ssh-key.pub - - - -__Execute the pipelines in `osdu-infrastructure`__ - -> These pipelines need to be executed to completion in the specific order. - -1. `infrastructure-central-resources` - - > For the first run of the pipeline approvals will need to be made for the 2 secure files and the Service Connection. - -```bash -# Create and Deploy the Pipeline -az pipelines create \ - --name 'infrastructure-central-resources' \ - --repository infra-azure-provisioning \ - --branch master \ - --repository-type tfsgit \ - --yaml-path /infra/templates/osdu-r3-mvp/pipeline-central-resources.yml \ - -ojson -``` - - -2. `infrastructure-data-partition` - - > For the first run of the pipeline approvals will need to be made for the 2 secure files and the Service Connection. - -```bash -# Create and Deploy the Pipeline -az pipelines create \ - --name 'infrastructure-data-partition' \ - --repository infra-azure-provisioning \ - --branch master \ - --repository-type tfsgit \ - --yaml-path /infra/templates/osdu-r3-mvp/pipeline-data-partition.yml \ - -ojson -``` - - -3. `azure-pipeline-service.yml` - - > For the first run of the pipeline approvals will need to be made for the 2 secure files and the Service Connection. - -```bash -# Create and Deploy the Pipeline -az pipelines create \ - --name 'infrastructure-service-resources' \ - --repository infra-azure-provisioning \ - --branch master \ - --repository-type tfsgit \ - --yaml-path /infra/templates/osdu-r3-mvp/pipeline-service-resources.yml \ - -ojson -``` - - - -## Deploy OSDU Services - -> This typically takes about 1-2 hours to complete. - - __Setup and Configure the ADO Library `Azure - OSDU`__ This variable group will be used to hold the common values for the services to be deployed and relate specifically to settings common to integration tests across all services. Variable values here that are other variables are being pulled from the Library group accessing the common Key Vault. @@ -674,44 +292,11 @@ az pipelines variable-group create \ -ojson ``` -__Load Storage Container Integration Test Data__ - -The data to be loaded before services are deployed and can be found in the osdu-infrastructure repository `osdu-infrastructure/docs/osdu/integration-test-data/`. - -Container: `legal-service-azure-configuration` - -- Legal_COO.json - - -__Load Cosmos DB Integration Test Data__ - -The data to be loaded before services are deployed and can be found in the osdu-infrastructure repository `osdu-infrastructure/docs/osdu/integration-test-data/` and has to be modified with environment specific information as necessary. -- tenant_info_1.json -- tenant_info_2.json -- user_info_1.json -- user_info_2.json -- legal_tag_1.json -- legal_tag_2.json -- legal_tag_3.json -- storage_schema_1.json -- storage_schema_2.json -- storage_schema_3.json -- storage_schema_4.json -- storage_schema_5.json -- storage_schema_6.json -- storage_schema_7.json -- storage_schema_8.json -- storage_schema_9.json -- storage_schema_10.json -- storage_schema_11.json - - -__Configure the ADO Charts and Service Pipelines__ +__Create the Chart Pipelines__ Create the pipelines and run things in this exact order. - 1. Add a Pipeline for __chart-osdu-common__ to deploy common components. > Ensure DNS is configured for your Gateway IP to DNS_HOST prior. @@ -740,7 +325,11 @@ Create the pipelines and run things in this exact order. _Validate:_ Authorization Policies exist in osdu namespace. -4. Add a Pipeline for __service-partition__ to deploy the Partition Service. +__Create the Service Pipelines__ + +Create the pipelines and run things in this exact order. + +1. Add a Pipeline for __service-partition__ to deploy the Partition Service. _Repo:_ `partition` @@ -749,8 +338,7 @@ Create the pipelines and run things in this exact order. _Validate:_ https:///api/partition/v1/swagger-ui.html is alive. - -5. Add a Pipeline for __service-entitlements-azure__ to deploy the Entitlements Service. +2. Add a Pipeline for __service-entitlements-azure__ to deploy the Entitlements Service. > This pipeline may have to be run twice for integration tests to pass due to a preload data issue. _Repo:_ `entitlements-azure` @@ -760,7 +348,7 @@ Create the pipelines and run things in this exact order. _Validate:_ https:///entitlements/v1/swagger-ui.html is alive. -6. Add a Pipeline for __service-legal__ to deploy the Legal Service. +3. Add a Pipeline for __service-legal__ to deploy the Legal Service. _Repo:_ `legal` @@ -769,7 +357,7 @@ Create the pipelines and run things in this exact order. _Validate:_ https:///api/legal/v1/swagger-ui.html is alive. -6. Add a Pipeline for __service-storage__ to deploy the Storage Service. +4. Add a Pipeline for __service-storage__ to deploy the Storage Service. _Repo:_ `storage` @@ -778,7 +366,7 @@ Create the pipelines and run things in this exact order. _Validate:_ https:///api/storage/v2/swagger-ui.html is alive. -7. Add a Pipeline for __service-indexer-queue__ to deploy the Indexer Queue Function. +5. Add a Pipeline for __service-indexer-queue__ to deploy the Indexer Queue Function. _Repo:_ `indexer-queue` @@ -787,7 +375,7 @@ Create the pipelines and run things in this exact order. _Validate:_ ScaledObject exist in osdu namespace. -8. Add a Pipeline for __service-indexer__ to deploy the Indexer Service. +6. Add a Pipeline for __service-indexer__ to deploy the Indexer Service. _Repo:_ `indexer-service` @@ -796,7 +384,7 @@ Create the pipelines and run things in this exact order. _Validate:_ https:///api/indexer/v2/swagger-ui.html is alive. -9. Add a Pipeline for __service-search__ to deploy the Search Service. +7. Add a Pipeline for __service-search__ to deploy the Search Service. _Repo:_ `search-service` diff --git a/tools/test_data/Legal_COO.json b/tools/test_data/Legal_COO.json new file mode 100644 index 00000000..586694a0 --- /dev/null +++ b/tools/test_data/Legal_COO.json @@ -0,0 +1,8 @@ +[ + { + "name": "Malaysia", + "alpha2": "MY", + "numeric": 458, + "residencyRisk": "Client consent required" + } +] diff --git a/tools/test_data/README.md b/tools/test_data/README.md new file mode 100644 index 00000000..779aef98 --- /dev/null +++ b/tools/test_data/README.md @@ -0,0 +1,78 @@ +# Test Data Upload Instructions + +## Getting started + +* [Python 2.7 or 3.5.3+][python] + +__SDK installation__ + +Install the Python SDK + +```bash +pip3 install azure-cosmos +``` + +__CLI Login__ + +Login to Azure CLI using the OSDU Environment Service Principal. + +```bash +# This logs your local Azure CLI in using the configured service principal. +az login --service-principal -u $ARM_CLIENT_ID -p $ARM_CLIENT_SECRET --tenant $ARM_TENANT_ID +``` + +__Upload Storage Blob Test Data__ + +This [file](../tools/test_data/Legal_COO.json) needs to be loaded into the Data Partition Storage Account in the container `legal-service-azure-configuration`. + +```bash +GROUP=$(az group list --query "[?contains(name, 'cr${UNIQUE}')].name" -otsv) +ENV_VAULT=$(az keyvault list --resource-group $GROUP --query [].name -otsv) +PARTITION_NAME=opendes + +az storage blob upload \ + --account-name $(az keyvault secret show --id https://${ENV_VAULT}.vault.azure.net/secrets/${PARTITION_NAME}-storage --query value -otsv) \ + --account-key $(az keyvault secret show --id https://${ENV_VAULT}.vault.azure.net/secrets/${PARTITION_NAME}-storage-key --query value -otsv) \ + --file tools/test_data/Legal_COO.json \ + --container-name legal-service-azure-configuration \ + --name Legal_COO.json +``` + +__Upload Cosmos DB Test Data__ + +These files need to be uploaded into the proper Cosmos Collections with the required values injected. + +- tenant_info_1.json +- tenant_info_2.json +- user_info_1.json +- user_info_2.json +- legal_tag_1.json +- legal_tag_2.json +- legal_tag_3.json +- storage_schema_1.json +- storage_schema_2.json +- storage_schema_3.json +- storage_schema_4.json +- storage_schema_5.json +- storage_schema_6.json +- storage_schema_7.json +- storage_schema_8.json +- storage_schema_9.json +- storage_schema_10.json +- storage_schema_11.json + + +```bash +# Retrieve Values from Common Key Vault +export NO_DATA_ACCESS_TESTER=$(az keyvault secret show --id https://$COMMON_VAULT.vault.azure.net/secrets/osdu-mvp-demo-noaccess-clientid --query value -otsv) + +# Retrieve Values from Environment Key Vault +export COSMOS_ENDPOINT=$(az keyvault secret show --id https://${ENV_VAULT}.vault.azure.net/secrets/${PARTITION_NAME}-cosmos-endpoint --query value -otsv) +export COSMOS_KEY=$(az keyvault secret show --id https://${ENV_VAULT}.vault.azure.net/secrets/${PARTITION_NAME}-cosmos-primary-key --query value -otsv) +export COSMOS_KEY=$(az keyvault secret show --id https://${ENV_VAULT}.vault.azure.net/secrets/${PARTITION_NAME}-cosmos-primary-key --query value -otsv) +export SERVICE_PRINCIPAL_ID=$(az keyvault secret show --id https://${ENV_VAULT}.vault.azure.net/secrets/app-dev-sp-username --query value -otsv) +export SERVICE_PRINCIPAL_OID=$(az keyvault secret show --id https://${ENV_VAULT}.vault.azure.net/secrets/app-dev-sp-id --query value -otsv) + +# Execute the Upload +python3 upload-data.py +``` diff --git a/tools/test_data/legal_tag_1.json b/tools/test_data/legal_tag_1.json new file mode 100644 index 00000000..04be9092 --- /dev/null +++ b/tools/test_data/legal_tag_1.json @@ -0,0 +1,21 @@ +{ + "id": "2146312565", + "legalTag": { + "id": 2146312565, + "name": "opendes-dps-integration-test-1566474656479", + "description": "", + "properties": { + "countryOfOrigin": [ + "US" + ], + "contractId": "A1234", + "expirationDate": 1570863500000, + "originator": "MyCompany", + "dataType": "Transferred Data", + "securityClassification": "Public", + "personalData": "No Personal Data", + "exportClassification": "EAR99" + }, + "isValid": false + } +} diff --git a/tools/test_data/legal_tag_2.json b/tools/test_data/legal_tag_2.json new file mode 100644 index 00000000..f86fcdf7 --- /dev/null +++ b/tools/test_data/legal_tag_2.json @@ -0,0 +1,21 @@ +{ + "id": "2066369556", + "legalTag": { + "id": 2066369556, + "name": "opendes-public-usa-dataset-7643990", + "description": "A legaltag used for demonstration purposes.", + "properties": { + "countryOfOrigin": [ + "US" + ], + "contractId": "No Contract Related", + "expirationDate": 4070822400000, + "originator": "SLB", + "dataType": "Public Domain Data", + "securityClassification": "Public", + "personalData": "No Personal Data", + "exportClassification": "Not - Technical Data" + }, + "isValid": true + } +} diff --git a/tools/test_data/legal_tag_3.json b/tools/test_data/legal_tag_3.json new file mode 100644 index 00000000..40c100e7 --- /dev/null +++ b/tools/test_data/legal_tag_3.json @@ -0,0 +1,21 @@ +{ + "id": "-760142234", + "legalTag": { + "id": -760142234, + "name": "opendes-dps-integration-test-valid2-legal-tag", + "description": "A legaltag used for Integration Testing.", + "properties": { + "countryOfOrigin": [ + "US" + ], + "contractId": "No Contract Related", + "expirationDate": 4070822400000, + "originator": "SLB", + "dataType": "Public Domain Data", + "securityClassification": "Public", + "personalData": "No Personal Data", + "exportClassification": "Not - Technical Data" + }, + "isValid": true + } +} diff --git a/tools/test_data/storage_schema_1.json b/tools/test_data/storage_schema_1.json new file mode 100644 index 00000000..588695a8 --- /dev/null +++ b/tools/test_data/storage_schema_1.json @@ -0,0 +1,68 @@ +{ + "kind": "opendes:testschema01:well:2.0.0", + "extension": null, + "user": "$SERVICE_PRINCIPAL_ID", + "schemaItems": [ + { + "path": "Field", + "kind": "string" + }, + { + "path": "Location", + "kind": "core:dl:geopoint:1.0.0" + }, + { + "path": "Basin", + "kind": "string" + }, + { + "path": "County", + "kind": "string" + }, + { + "path": "State", + "kind": "string" + }, + { + "path": "Country", + "kind": "string" + }, + { + "path": "WellStatus", + "kind": "string" + }, + { + "path": "OriginalOperator", + "kind": "string" + }, + { + "path": "WellName", + "kind": "string" + }, + { + "path": "WellType", + "kind": "string" + }, + { + "path": "EmptyAttribute", + "kind": "string" + }, + { + "path": "Rank", + "kind": "int" + }, + { + "path": "Score", + "kind": "int" + }, + { + "path": "Established", + "kind": "datetime" + }, + { + "path": "InvalidInteger", + "kind": "int" + } + ], + "id": "opendes:testschema01:well:2.0.0" +} diff --git a/tools/test_data/storage_schema_10.json b/tools/test_data/storage_schema_10.json new file mode 100644 index 00000000..e1ec80ea --- /dev/null +++ b/tools/test_data/storage_schema_10.json @@ -0,0 +1,68 @@ +{ + "kind": "opendes:testschema10:well:2.0.0", + "extension": null, + "user": "$SERVICE_PRINCIPAL_ID", + "schemaItems": [ + { + "path": "Field", + "kind": "string" + }, + { + "path": "Location", + "kind": "core:dl:geopoint:1.0.0" + }, + { + "path": "Basin", + "kind": "string" + }, + { + "path": "County", + "kind": "string" + }, + { + "path": "State", + "kind": "string" + }, + { + "path": "Country", + "kind": "string" + }, + { + "path": "WellStatus", + "kind": "string" + }, + { + "path": "OriginalOperator", + "kind": "string" + }, + { + "path": "WellName", + "kind": "string" + }, + { + "path": "WellType", + "kind": "string" + }, + { + "path": "EmptyAttribute", + "kind": "string" + }, + { + "path": "Rank", + "kind": "int" + }, + { + "path": "Score", + "kind": "int" + }, + { + "path": "Established", + "kind": "datetime" + }, + { + "path": "InvalidInteger", + "kind": "int" + } + ], + "id": "opendes:testschema10:well:2.0.0" +} diff --git a/tools/test_data/storage_schema_11.json b/tools/test_data/storage_schema_11.json new file mode 100644 index 00000000..99d1a04c --- /dev/null +++ b/tools/test_data/storage_schema_11.json @@ -0,0 +1,68 @@ +{ + "kind": "opendes:testschema11:well:2.0.0", + "extension": null, + "user": "$SERVICE_PRINCIPAL_ID", + "schemaItems": [ + { + "path": "Field", + "kind": "string" + }, + { + "path": "Location", + "kind": "core:dl:geopoint:1.0.0" + }, + { + "path": "Basin", + "kind": "string" + }, + { + "path": "County", + "kind": "string" + }, + { + "path": "State", + "kind": "string" + }, + { + "path": "Country", + "kind": "string" + }, + { + "path": "WellStatus", + "kind": "string" + }, + { + "path": "OriginalOperator", + "kind": "string" + }, + { + "path": "WellName", + "kind": "string" + }, + { + "path": "WellType", + "kind": "string" + }, + { + "path": "EmptyAttribute", + "kind": "string" + }, + { + "path": "Rank", + "kind": "int" + }, + { + "path": "Score", + "kind": "int" + }, + { + "path": "Established", + "kind": "datetime" + }, + { + "path": "InvalidInteger", + "kind": "int" + } + ], + "id": "opendes:testschema11:well:2.0.0" +} diff --git a/tools/test_data/storage_schema_2.json b/tools/test_data/storage_schema_2.json new file mode 100644 index 00000000..948e493c --- /dev/null +++ b/tools/test_data/storage_schema_2.json @@ -0,0 +1,68 @@ +{ + "kind": "opendes:testschema02:well:2.0.0", + "extension": null, + "user": "$SERVICE_PRINCIPAL_ID", + "schemaItems": [ + { + "path": "Field", + "kind": "string" + }, + { + "path": "Location", + "kind": "core:dl:geopoint:1.0.0" + }, + { + "path": "Basin", + "kind": "string" + }, + { + "path": "County", + "kind": "string" + }, + { + "path": "State", + "kind": "string" + }, + { + "path": "Country", + "kind": "string" + }, + { + "path": "WellStatus", + "kind": "string" + }, + { + "path": "OriginalOperator", + "kind": "string" + }, + { + "path": "WellName", + "kind": "string" + }, + { + "path": "WellType", + "kind": "string" + }, + { + "path": "EmptyAttribute", + "kind": "string" + }, + { + "path": "Rank", + "kind": "int" + }, + { + "path": "Score", + "kind": "int" + }, + { + "path": "Established", + "kind": "datetime" + }, + { + "path": "InvalidInteger", + "kind": "int" + } + ], + "id": "opendes:testschema02:well:2.0.0" +} diff --git a/tools/test_data/storage_schema_3.json b/tools/test_data/storage_schema_3.json new file mode 100644 index 00000000..f538ad16 --- /dev/null +++ b/tools/test_data/storage_schema_3.json @@ -0,0 +1,68 @@ +{ + "kind": "opendes:testschema03:well:2.0.0", + "extension": null, + "user": "$SERVICE_PRINCIPAL_ID", + "schemaItems": [ + { + "path": "Field", + "kind": "string" + }, + { + "path": "Location", + "kind": "core:dl:geopoint:1.0.0" + }, + { + "path": "Basin", + "kind": "string" + }, + { + "path": "County", + "kind": "string" + }, + { + "path": "State", + "kind": "string" + }, + { + "path": "Country", + "kind": "string" + }, + { + "path": "WellStatus", + "kind": "string" + }, + { + "path": "OriginalOperator", + "kind": "string" + }, + { + "path": "WellName", + "kind": "string" + }, + { + "path": "WellType", + "kind": "string" + }, + { + "path": "EmptyAttribute", + "kind": "string" + }, + { + "path": "Rank", + "kind": "int" + }, + { + "path": "Score", + "kind": "int" + }, + { + "path": "Established", + "kind": "datetime" + }, + { + "path": "InvalidInteger", + "kind": "int" + } + ], + "id": "opendes:testschema03:well:2.0.0" +} diff --git a/tools/test_data/storage_schema_4.json b/tools/test_data/storage_schema_4.json new file mode 100644 index 00000000..fa1c61c7 --- /dev/null +++ b/tools/test_data/storage_schema_4.json @@ -0,0 +1,68 @@ +{ + "kind": "opendes:testschema04:well:2.0.0", + "extension": null, + "user": "$SERVICE_PRINCIPAL_ID", + "schemaItems": [ + { + "path": "Field", + "kind": "string" + }, + { + "path": "Location", + "kind": "core:dl:geopoint:1.0.0" + }, + { + "path": "Basin", + "kind": "string" + }, + { + "path": "County", + "kind": "string" + }, + { + "path": "State", + "kind": "string" + }, + { + "path": "Country", + "kind": "string" + }, + { + "path": "WellStatus", + "kind": "string" + }, + { + "path": "OriginalOperator", + "kind": "string" + }, + { + "path": "WellName", + "kind": "string" + }, + { + "path": "WellType", + "kind": "string" + }, + { + "path": "EmptyAttribute", + "kind": "string" + }, + { + "path": "Rank", + "kind": "int" + }, + { + "path": "Score", + "kind": "int" + }, + { + "path": "Established", + "kind": "datetime" + }, + { + "path": "InvalidInteger", + "kind": "int" + } + ], + "id": "opendes:testschema04:well:2.0.0" +} diff --git a/tools/test_data/storage_schema_5.json b/tools/test_data/storage_schema_5.json new file mode 100644 index 00000000..91314e65 --- /dev/null +++ b/tools/test_data/storage_schema_5.json @@ -0,0 +1,68 @@ +{ + "kind": "opendes:testschema05:well:2.0.0", + "extension": null, + "user": "$SERVICE_PRINCIPAL_ID", + "schemaItems": [ + { + "path": "Field", + "kind": "string" + }, + { + "path": "Location", + "kind": "core:dl:geopoint:1.0.0" + }, + { + "path": "Basin", + "kind": "string" + }, + { + "path": "County", + "kind": "string" + }, + { + "path": "State", + "kind": "string" + }, + { + "path": "Country", + "kind": "string" + }, + { + "path": "WellStatus", + "kind": "string" + }, + { + "path": "OriginalOperator", + "kind": "string" + }, + { + "path": "WellName", + "kind": "string" + }, + { + "path": "WellType", + "kind": "string" + }, + { + "path": "EmptyAttribute", + "kind": "string" + }, + { + "path": "Rank", + "kind": "int" + }, + { + "path": "Score", + "kind": "int" + }, + { + "path": "Established", + "kind": "datetime" + }, + { + "path": "InvalidInteger", + "kind": "int" + } + ], + "id": "opendes:testschema05:well:2.0.0" +} diff --git a/tools/test_data/storage_schema_6.json b/tools/test_data/storage_schema_6.json new file mode 100644 index 00000000..0999785f --- /dev/null +++ b/tools/test_data/storage_schema_6.json @@ -0,0 +1,68 @@ +{ + "kind": "opendes:testschema06:well:2.0.0", + "extension": null, + "user": "$SERVICE_PRINCIPAL_ID", + "schemaItems": [ + { + "path": "Field", + "kind": "string" + }, + { + "path": "Location", + "kind": "core:dl:geopoint:1.0.0" + }, + { + "path": "Basin", + "kind": "string" + }, + { + "path": "County", + "kind": "string" + }, + { + "path": "State", + "kind": "string" + }, + { + "path": "Country", + "kind": "string" + }, + { + "path": "WellStatus", + "kind": "string" + }, + { + "path": "OriginalOperator", + "kind": "string" + }, + { + "path": "WellName", + "kind": "string" + }, + { + "path": "WellType", + "kind": "string" + }, + { + "path": "EmptyAttribute", + "kind": "string" + }, + { + "path": "Rank", + "kind": "int" + }, + { + "path": "Score", + "kind": "int" + }, + { + "path": "Established", + "kind": "datetime" + }, + { + "path": "InvalidInteger", + "kind": "int" + } + ], + "id": "opendes:testschema06:well:2.0.0" +} diff --git a/tools/test_data/storage_schema_7.json b/tools/test_data/storage_schema_7.json new file mode 100644 index 00000000..0a4b039c --- /dev/null +++ b/tools/test_data/storage_schema_7.json @@ -0,0 +1,68 @@ +{ + "kind": "opendes:testschema07:well:2.0.0", + "extension": null, + "user": "$SERVICE_PRINCIPAL_ID", + "schemaItems": [ + { + "path": "Field", + "kind": "string" + }, + { + "path": "Location", + "kind": "core:dl:geopoint:1.0.0" + }, + { + "path": "Basin", + "kind": "string" + }, + { + "path": "County", + "kind": "string" + }, + { + "path": "State", + "kind": "string" + }, + { + "path": "Country", + "kind": "string" + }, + { + "path": "WellStatus", + "kind": "string" + }, + { + "path": "OriginalOperator", + "kind": "string" + }, + { + "path": "WellName", + "kind": "string" + }, + { + "path": "WellType", + "kind": "string" + }, + { + "path": "EmptyAttribute", + "kind": "string" + }, + { + "path": "Rank", + "kind": "int" + }, + { + "path": "Score", + "kind": "int" + }, + { + "path": "Established", + "kind": "datetime" + }, + { + "path": "InvalidInteger", + "kind": "int" + } + ], + "id": "opendes:testschema07:well:2.0.0" +} diff --git a/tools/test_data/storage_schema_8.json b/tools/test_data/storage_schema_8.json new file mode 100644 index 00000000..3aed92eb --- /dev/null +++ b/tools/test_data/storage_schema_8.json @@ -0,0 +1,68 @@ +{ + "kind": "opendes:testschema08:well:2.0.0", + "extension": null, + "user": "$SERVICE_PRINCIPAL_ID", + "schemaItems": [ + { + "path": "Field", + "kind": "string" + }, + { + "path": "Location", + "kind": "core:dl:geopoint:1.0.0" + }, + { + "path": "Basin", + "kind": "string" + }, + { + "path": "County", + "kind": "string" + }, + { + "path": "State", + "kind": "string" + }, + { + "path": "Country", + "kind": "string" + }, + { + "path": "WellStatus", + "kind": "string" + }, + { + "path": "OriginalOperator", + "kind": "string" + }, + { + "path": "WellName", + "kind": "string" + }, + { + "path": "WellType", + "kind": "string" + }, + { + "path": "EmptyAttribute", + "kind": "string" + }, + { + "path": "Rank", + "kind": "int" + }, + { + "path": "Score", + "kind": "int" + }, + { + "path": "Established", + "kind": "datetime" + }, + { + "path": "InvalidInteger", + "kind": "int" + } + ], + "id": "opendes:testschema08:well:2.0.0" +} diff --git a/tools/test_data/storage_schema_9.json b/tools/test_data/storage_schema_9.json new file mode 100644 index 00000000..e2ebb8ce --- /dev/null +++ b/tools/test_data/storage_schema_9.json @@ -0,0 +1,68 @@ +{ + "kind": "opendes:testschema09:well:2.0.0", + "extension": null, + "user": "$SERVICE_PRINCIPAL_ID", + "schemaItems": [ + { + "path": "Field", + "kind": "string" + }, + { + "path": "Location", + "kind": "core:dl:geopoint:1.0.0" + }, + { + "path": "Basin", + "kind": "string" + }, + { + "path": "County", + "kind": "string" + }, + { + "path": "State", + "kind": "string" + }, + { + "path": "Country", + "kind": "string" + }, + { + "path": "WellStatus", + "kind": "string" + }, + { + "path": "OriginalOperator", + "kind": "string" + }, + { + "path": "WellName", + "kind": "string" + }, + { + "path": "WellType", + "kind": "string" + }, + { + "path": "EmptyAttribute", + "kind": "string" + }, + { + "path": "Rank", + "kind": "int" + }, + { + "path": "Score", + "kind": "int" + }, + { + "path": "Established", + "kind": "datetime" + }, + { + "path": "InvalidInteger", + "kind": "int" + } + ], + "id": "opendes:testschema09:well:2.0.0" +} diff --git a/tools/test_data/tenant_info_1.json b/tools/test_data/tenant_info_1.json new file mode 100644 index 00000000..410e2b48 --- /dev/null +++ b/tools/test_data/tenant_info_1.json @@ -0,0 +1,24 @@ +{ + "id": "opendes", + "complianceRuleSet": "shared", + "serviceprincipalAppId": "$SERVICE_PRINCIPAL_ID", + "groups": [ + "service.storage.admin", + "service.legal.admin", + "data.datalake.viewer", + "data.test1", + "data.datalake.viewer", + "data.default.viewer", + "data.default.owner", + "service.search.admin", + "service.search.user", + "data.default.viewers", + "data.default.owners", + "service.schema-service.viewers", + "service.schema-service.editors", + "service.entitlements.admin", + "service.workflow.creator", + "service.workflow.viewer", + "service.workflow.admin" + ] +} diff --git a/tools/test_data/tenant_info_2.json b/tools/test_data/tenant_info_2.json new file mode 100644 index 00000000..9d431744 --- /dev/null +++ b/tools/test_data/tenant_info_2.json @@ -0,0 +1,23 @@ +{ + "id": "common", + "complianceRuleSet": "shared", + "serviceprincipalAppId": "$SERVICE_PRINCIPAL_ID", + "groups": [ + "service.storage.admin", + "service.legal.admin", + "data.datalake.admin", + "data.datalake.viewer", + "data.default.viewer", + "data.default.owner", + "service.search.admin", + "service.search.user", + "data.default.viewers", + "data.default.owners", + "service.schema-service.viewers", + "service.schema-service.editors", + "service.entitlements.admin", + "service.workflow.creator", + "service.workflow.viewer", + "service.workflow.admin" + ] +} diff --git a/tools/test_data/upload-data.py b/tools/test_data/upload-data.py new file mode 100755 index 00000000..2d68164c --- /dev/null +++ b/tools/test_data/upload-data.py @@ -0,0 +1,135 @@ +# Copyright © Microsoft Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +from azure.cosmos import CosmosClient, PartitionKey +import json +import pprint +import os +import time + +cosmos_url = os.environ['COSMOS_ENDPOINT'] +cosmos_key = os.environ['COSMOS_KEY'] +service_principal_id = os.environ['SERVICE_PRINCIPAL_ID'] +service_principal_oid = os.environ['SERVICE_PRINCIPAL_OID'] +no_data_access_tester = os.environ['NO_DATA_ACCESS_TESTER'] + +cosmos_client = CosmosClient(cosmos_url, credential = cosmos_key) +db = cosmos_client.get_database_client('osdu-db') + + +def tenant_info(db): + container = db.get_container_client("TenantInfo") + + + with open("./tenant_info_1.json", "r") as f : + tenant_info_1 = json.load(f) + with open("./tenant_info_2.json", "r") as f : + tenant_info_2 = json.load(f) + + tenant_info_1["serviceprincipalAppId"] = service_principal_id + tenant_info_2["serviceprincipalAppId"] = service_principal_id + + container.upsert_item(tenant_info_1) + container.upsert_item(tenant_info_2) + return + +def user_info(db): + container = db.get_container_client("UserInfo") + + with open("./user_info_1.json", "r") as f: + user_info_1 = json.load(f) + with open("./user_info_2.json", "r") as f : + user_info_2 = json.load(f) + + user_info_1["id"] = service_principal_oid + user_info_1["uid"] = service_principal_id + user_info_2["id"] = no_data_access_tester + + container.upsert_item(user_info_1) + container.upsert_item(user_info_2) + return + +def legal_tag(db): + container = db.get_container_client("LegalTag") + + with open("./legal_tag_1.json", "r") as f: + legal_tag_1 = json.load(f) + with open("./legal_tag_2.json", "r") as f: + legal_tag_2 = json.load(f) + with open("./legal_tag_3.json", "r") as f: + legal_tag_3 = json.load(f) + + container.upsert_item(legal_tag_1) + container.upsert_item(legal_tag_2) + container.upsert_item(legal_tag_3) + return + +def storage_schema(db): + container = db.get_container_client("StorageSchema") + + with open("./storage_schema_1.json", "r") as f: + storage_schema_1 = json.load(f) + with open("./storage_schema_2.json", "r") as f: + storage_schema_2 = json.load(f) + with open("./storage_schema_3.json", "r") as f: + storage_schema_3 = json.load(f) + with open("./storage_schema_4.json", "r") as f: + storage_schema_4 = json.load(f) + with open("./storage_schema_5.json", "r") as f: + storage_schema_5 = json.load(f) + with open("./storage_schema_6.json", "r") as f: + storage_schema_6 = json.load(f) + with open("./storage_schema_7.json", "r") as f: + storage_schema_7 = json.load(f) + with open("./storage_schema_8.json", "r") as f: + storage_schema_8 = json.load(f) + with open("./storage_schema_9.json", "r") as f: + storage_schema_9 = json.load(f) + with open("./storage_schema_10.json", "r") as f: + storage_schema_10 = json.load(f) + with open("./storage_schema_11.json", "r") as f: + storage_schema_11 = json.load(f) + + storage_schema_1["user"] = service_principal_id + storage_schema_2["user"] = service_principal_id + storage_schema_3["user"] = service_principal_id + storage_schema_4["user"] = service_principal_id + storage_schema_5["user"] = service_principal_id + storage_schema_6["user"] = service_principal_id + storage_schema_7["user"] = service_principal_id + storage_schema_8["user"] = service_principal_id + storage_schema_9["user"] = service_principal_id + storage_schema_10["user"] = service_principal_id + storage_schema_11["user"] = service_principal_id + + container.upsert_item(storage_schema_1) + container.upsert_item(storage_schema_2) + container.upsert_item(storage_schema_3) + container.upsert_item(storage_schema_4) + container.upsert_item(storage_schema_5) + container.upsert_item(storage_schema_6) + container.upsert_item(storage_schema_7) + container.upsert_item(storage_schema_8) + container.upsert_item(storage_schema_9) + container.upsert_item(storage_schema_10) + container.upsert_item(storage_schema_11) + return + + +if __name__ == "__main__": + tenant_info(db) + user_info(db) + legal_tag(db) + storage_schema(db) diff --git a/tools/test_data/user_info_1.json b/tools/test_data/user_info_1.json new file mode 100644 index 00000000..aad2fba6 --- /dev/null +++ b/tools/test_data/user_info_1.json @@ -0,0 +1,49 @@ +{ + "id": "$SERVICE_PRINCIPAL_OBJECT_ID", + "uid": "$SERVICE_PRINCIPAL_APPLICATION_ID", + "tenants": [ + { + "name": "common", + "groups": [ + "service.storage.admin", + "service.legal.admin", + "data.datalake.admin", + "data.datalake.viewer", + "data.default.viewer", + "data.default.owner", + "service.search.admin", + "service.search.user", + "data.default.viewers", + "data.default.owners", + "service.entitlements.admin", + "service.schema-service.viewers", + "service.schema-service.editors", + "service.workflow.creator", + "service.workflow.viewer", + "service.workflow.admin" + ] + }, + { + "name": "opendes", + "groups": [ + "service.storage.admin", + "service.legal.admin", + "data.datalake.viewer", + "data.test1", + "data.datalake.viewer", + "data.default.viewer", + "data.default.owner", + "service.search.admin", + "service.search.user", + "data.default.viewers", + "data.default.owners", + "service.entitlements.admin", + "service.schema-service.viewers", + "service.schema-service.editors", + "service.workflow.creator", + "service.workflow.viewer", + "service.workflow.admin" + ] + } + ] +} diff --git a/tools/test_data/user_info_2.json b/tools/test_data/user_info_2.json new file mode 100644 index 00000000..416ea7b6 --- /dev/null +++ b/tools/test_data/user_info_2.json @@ -0,0 +1,20 @@ +{ + "id": "$SERVICE_PRINCIPAL_NO_ACCESS_APP_ID", + "uid": "", + "tenants": [ + { + "name": "common", + "groups": [ + "service.storage.admin", + "service.legal.admin" + ] + }, + { + "name": "opendes", + "groups": [ + "service.storage.admin", + "service.legal.admin" + ] + } + ] +} -- GitLab From 73f315134e525c757248a8c0730dfe8ba045070b Mon Sep 17 00:00:00 2001 From: Daniel Scholl Date: Tue, 20 Oct 2020 15:55:21 -0500 Subject: [PATCH 5/9] Documentation Updates --- README.md | 6 +++--- docs/code-mirroring.md | 2 -- docs/dns-setup.md | 9 +++++++++ docs/infra-automation.md | 2 -- docs/service-automation.md | 4 +--- infra/README.md | 15 --------------- infra/templates/osdu-r3-mvp/README.md | 16 ---------------- .../osdu-r3-mvp/central_resources/README.md | 17 ----------------- .../osdu-r3-mvp/data_partition/README.md | 17 ----------------- .../osdu-r3-mvp/service_resources/README.md | 17 ----------------- 10 files changed, 13 insertions(+), 92 deletions(-) create mode 100644 docs/dns-setup.md diff --git a/README.md b/README.md index 50478b29..95f87a13 100644 --- a/README.md +++ b/README.md @@ -237,14 +237,14 @@ __Manual Installation__ 1. Install the Infrastructure following directions [here](./infra/templates/osdu-r3-mvp). -2. Manually configure your DNS_HOST to the IP Address of the environment IP Address. +2. Setup DNS to point to the deployed infrastructure following directions [here](./docs/dns-setup.md). 3. Upload the Integration Test Data following directions [here](./tools/test_data). 3. Deploy the application helm charts following the directions [here](./charts). -## Automated Pipeline Installation +__Automated Pipeline Installation__ > This typically takes about 3 hours to complete. @@ -252,7 +252,7 @@ __Manual Installation__ 2. Setup Infrastructure Automation following directions [here](./docs/infra-automation.md). -3. Manually configure your DNS_HOST to the IP Address of the environment IP Address. +2. Setup DNS to point to the deployed infrastructure following directions [here](./docs/dns-setup.md). 4. Upload the Integration Test Data following directions [here](./tools/test_data). diff --git a/docs/code-mirroring.md b/docs/code-mirroring.md index 4c65083c..37f75c6a 100644 --- a/docs/code-mirroring.md +++ b/docs/code-mirroring.md @@ -1,7 +1,5 @@ # Setup Mirroring for Gitlab Repositories -> This typically takes about 10 minutes to complete. - __Create Empty Repositories__ Empty repositories need to be created that will be used by a pipeline to mirror gitlab repositories into. diff --git a/docs/dns-setup.md b/docs/dns-setup.md new file mode 100644 index 00000000..89813c56 --- /dev/null +++ b/docs/dns-setup.md @@ -0,0 +1,9 @@ +# DNS Setup + +Manually update your DNS A Records to point to the Public IP Address for the environment. + +```bash +# Get IP Address +RESOURCE_GROUP=$(az group list --query "[?contains(name, '${UNIQUE}sr')].name" -otsv |grep -v MC) +az network public-ip list --resource-group $RESOURCE_GROUP --query [].ipAddress -otsv +``` diff --git a/docs/infra-automation.md b/docs/infra-automation.md index 6644db6f..c8f5830d 100644 --- a/docs/infra-automation.md +++ b/docs/infra-automation.md @@ -1,7 +1,5 @@ # Deploy Infrastructure -> This typically takes about 2 hours to complete. - __Configure Azure DevOps Service Connection__ - Configure an [ARM Resources Service Connection](https://docs.microsoft.com/en-us/azure/devops/pipelines/library/connect-to-azure?view=azure-devops) for the desired subscription. diff --git a/docs/service-automation.md b/docs/service-automation.md index be9ed5c2..a183fab2 100644 --- a/docs/service-automation.md +++ b/docs/service-automation.md @@ -1,6 +1,4 @@ -## Create OSDU Service Libraries - -> This typically takes about 10 minutes to complete. +# Create OSDU Service Libraries __Setup and Configure the ADO Library `Azure - OSDU`__ diff --git a/infra/README.md b/infra/README.md index 993d2b7e..88edd836 100644 --- a/infra/README.md +++ b/infra/README.md @@ -60,18 +60,3 @@ $ tree -d ├── integration └── unit ``` - -## License -Copyright © Microsoft Corporation - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - -[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0) - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. diff --git a/infra/templates/osdu-r3-mvp/README.md b/infra/templates/osdu-r3-mvp/README.md index c24369d0..d4195c5b 100644 --- a/infra/templates/osdu-r3-mvp/README.md +++ b/infra/templates/osdu-r3-mvp/README.md @@ -15,19 +15,3 @@ Follow the directions in the [`data_resources`](./data_partition/README.md) envi __Deploy Service Resources__ Follow the directions in the [`service_resources`](./service_resources/README.md) environment. - -## License - -Copyright © Microsoft Corporation - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - -[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0) - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. diff --git a/infra/templates/osdu-r3-mvp/central_resources/README.md b/infra/templates/osdu-r3-mvp/central_resources/README.md index a2a4bc43..ddced8c2 100644 --- a/infra/templates/osdu-r3-mvp/central_resources/README.md +++ b/infra/templates/osdu-r3-mvp/central_resources/README.md @@ -127,20 +127,3 @@ Integration tests can be run using the following command: ``` go test -v $(go list ./... | grep "integration") ``` - - -## License - -Copyright © Microsoft Corporation - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - -[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0) - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. diff --git a/infra/templates/osdu-r3-mvp/data_partition/README.md b/infra/templates/osdu-r3-mvp/data_partition/README.md index 443ad8a8..9f64acc3 100644 --- a/infra/templates/osdu-r3-mvp/data_partition/README.md +++ b/infra/templates/osdu-r3-mvp/data_partition/README.md @@ -95,20 +95,3 @@ Integration tests can be run using the following command: ``` go test -v $(go list ./... | grep "integration") ``` - - -## License - -Copyright © Microsoft Corporation - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - -[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0) - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. diff --git a/infra/templates/osdu-r3-mvp/service_resources/README.md b/infra/templates/osdu-r3-mvp/service_resources/README.md index 701fdc1d..ca5b3a00 100644 --- a/infra/templates/osdu-r3-mvp/service_resources/README.md +++ b/infra/templates/osdu-r3-mvp/service_resources/README.md @@ -94,20 +94,3 @@ Integration tests can be run using the following command: ``` go test -v $(go list ./... | grep "integration") ``` - - -## License - -Copyright © Microsoft Corporation - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - -[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0) - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -- GitLab From 263f53279f020d818b16086d2f1698fc71e58154 Mon Sep 17 00:00:00 2001 From: Daniel Scholl Date: Tue, 20 Oct 2020 15:58:05 -0500 Subject: [PATCH 6/9] Updated Readme --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 95f87a13..160add6b 100644 --- a/README.md +++ b/README.md @@ -243,6 +243,8 @@ __Manual Installation__ 3. Deploy the application helm charts following the directions [here](./charts). +5. Setup Environment Variables for IDE Development and Integration Testing. + __Automated Pipeline Installation__ -- GitLab From c9a8b136796a10167bd09a3a46d70ccf0fb97178 Mon Sep 17 00:00:00 2001 From: Daniel Scholl Date: Tue, 20 Oct 2020 19:17:44 -0500 Subject: [PATCH 7/9] Added Pipeline --- devops/pipelines/chart-osdu-common.yml | 54 ++++++++++++++++++ devops/pipelines/chart-osdu-istio-auth.yml | 55 ++++++++++++++++++ devops/pipelines/chart-osdu-istio.yml | 55 ++++++++++++++++++ .../infrastructure-central-resources.yml | 54 ++++++++++++++++++ .../infrastructure-data-partition.yml | 56 +++++++++++++++++++ .../infrastructure-service-resources.yml | 55 ++++++++++++++++++ docs/service-automation.md | 12 +++- tools/rest/auth_token.http | 2 +- 8 files changed, 339 insertions(+), 4 deletions(-) create mode 100644 devops/pipelines/chart-osdu-common.yml create mode 100644 devops/pipelines/chart-osdu-istio-auth.yml create mode 100644 devops/pipelines/chart-osdu-istio.yml create mode 100644 devops/pipelines/infrastructure-central-resources.yml create mode 100644 devops/pipelines/infrastructure-data-partition.yml create mode 100644 devops/pipelines/infrastructure-service-resources.yml diff --git a/devops/pipelines/chart-osdu-common.yml b/devops/pipelines/chart-osdu-common.yml new file mode 100644 index 00000000..b7ad624c --- /dev/null +++ b/devops/pipelines/chart-osdu-common.yml @@ -0,0 +1,54 @@ +# Copyright © Microsoft Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +trigger: + batch: true + branches: + include: + - master + paths: + include: + - /charts/osdu-common/* + exclude: + - /**/*.md + +resources: + repositories: + - repository: FluxRepo + type: git + name: k8-gitops-manifests + +variables: + - group: 'Azure - OSDU' + - group: 'Azure - OSDU secrets' + + - name: serviceName + value: "osdu-common" + - name: chartPath + value: "charts/osdu-common" + - name: valuesFile + value: "charts/helm-config.yaml" + - name: 'MANIFEST_REPO' + value: $[ resources.repositories['FluxRepo'].name ] + +stages: + - template: /devops/chart-stages.yml + parameters: + serviceName: ${{ variables.serviceName }} + chartPath: ${{ variables.chartPath }} + valuesFile: ${{ variables.valuesFile }} + skipDeploy: ${{ variables.SKIP_DEPLOY }} + providers: + - name: Azure + environments: ["demo"] diff --git a/devops/pipelines/chart-osdu-istio-auth.yml b/devops/pipelines/chart-osdu-istio-auth.yml new file mode 100644 index 00000000..26597c77 --- /dev/null +++ b/devops/pipelines/chart-osdu-istio-auth.yml @@ -0,0 +1,55 @@ +# Copyright © Microsoft Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + trigger: + batch: true + branches: + include: + - master + paths: + include: + - /charts/osdu-istio-auth/* + exclude: + - /**/*.md + + resources: + repositories: + - repository: FluxRepo + type: git + name: k8-gitops-manifests + + variables: + - group: 'Azure - OSDU' + - group: 'Azure - OSDU secrets' + + - name: serviceName + value: "osdu-istio-auth" + - name: chartPath + value: "charts/osdu-istio-auth" + - name: valuesFile + value: "charts/helm-config.yaml" + - name: 'MANIFEST_REPO' + value: $[ resources.repositories['FluxRepo'].name ] + + stages: + - template: /devops/chart-stages.yml + parameters: + serviceName: ${{ variables.serviceName }} + chartPath: ${{ variables.chartPath }} + valuesFile: ${{ variables.valuesFile }} + skipDeploy: ${{ variables.SKIP_DEPLOY }} + skipCheck: true + providers: + - name: Azure + environments: ["demo"] diff --git a/devops/pipelines/chart-osdu-istio.yml b/devops/pipelines/chart-osdu-istio.yml new file mode 100644 index 00000000..092c45a9 --- /dev/null +++ b/devops/pipelines/chart-osdu-istio.yml @@ -0,0 +1,55 @@ +# Copyright © Microsoft Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + trigger: + batch: true + branches: + include: + - master + paths: + include: + - /charts/osdu-istio/* + exclude: + - /**/*.md + + resources: + repositories: + - repository: FluxRepo + type: git + name: k8-gitops-manifests + + variables: + - group: 'Azure - OSDU' + - group: 'Azure - OSDU secrets' + + - name: serviceName + value: "osdu-istio" + - name: chartPath + value: "charts/osdu-istio" + - name: valuesFile + value: "charts/helm-config.yaml" + - name: 'MANIFEST_REPO' + value: $[ resources.repositories['FluxRepo'].name ] + + stages: + - template: /devops/chart-stages.yml + parameters: + serviceName: ${{ variables.serviceName }} + chartPath: ${{ variables.chartPath }} + valuesFile: ${{ variables.valuesFile }} + skipDeploy: ${{ variables.SKIP_DEPLOY }} + skipCheck: true + providers: + - name: Azure + environments: ["demo"] diff --git a/devops/pipelines/infrastructure-central-resources.yml b/devops/pipelines/infrastructure-central-resources.yml new file mode 100644 index 00000000..6a18163f --- /dev/null +++ b/devops/pipelines/infrastructure-central-resources.yml @@ -0,0 +1,54 @@ +# Copyright © Microsoft Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +trigger: + batch: true + branches: + include: + - master + paths: + include: + - /infra/modules/providers/* + - /infra/templates/osdu-r3-mvp/central_resources/* + - /test-harness/terratest-extensions/* + exclude: + - /**/*.md + + +variables: +- name: 'TF_VERSION' + value: '0.12.29' +- name: 'GO_VERSION' + value: '1.12.14' +- name: 'TF_ROOT_DIR' + value: 'infra' +- name: 'PIPELINE_ROOT_DIR' + value: 'devops' +- name: 'TEST_HARNESS_DIR' + value: 'test-harness' +- group: 'Infrastructure Pipeline Variables' + +stages: +- template: /devops/infra-prepare-stage.yml + parameters: + environments: + - name: 'demo' + enablePrIsolation: false + resourceNameIsolationLevel: 8 + + configurations: + - jobName: central_resources + terraformWorkspacePrefix: cr + terraformTemplatePath: /infra/templates/osdu-r3-mvp/central_resources + deploymentTimeoutInMinutes: 120 diff --git a/devops/pipelines/infrastructure-data-partition.yml b/devops/pipelines/infrastructure-data-partition.yml new file mode 100644 index 00000000..c256a6e9 --- /dev/null +++ b/devops/pipelines/infrastructure-data-partition.yml @@ -0,0 +1,56 @@ +# Copyright © Microsoft Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +trigger: + batch: true + branches: + include: + - master + paths: + include: + - /infra/modules/providers/* + - /infra/templates/osdu-r3-mvp/data_partition/* + - /test-harness/terratest-extensions/* + exclude: + - /**/*.md + + +variables: +- name: 'TF_VERSION' + value: '0.12.29' +- name: 'GO_VERSION' + value: '1.12.14' +- name: 'TF_ROOT_DIR' + value: 'infra' +- name: 'PIPELINE_ROOT_DIR' + value: 'devops' +- name: 'TEST_HARNESS_DIR' + value: 'test-harness' +- group: 'Infrastructure Pipeline Variables' + +stages: +- template: /devops/infra-prepare-stage.yml + parameters: + environments: + - name: 'demo' + enablePrIsolation: false + resourceNameIsolationLevel: 8 + + configurations: + - jobName: data_partition_1 + terraformWorkspacePrefix: dp1 + terraformTemplatePath: /infra/templates/osdu-r3-mvp/data_partition + deploymentTimeoutInMinutes: 120 + dataPartitionName: opendes diff --git a/devops/pipelines/infrastructure-service-resources.yml b/devops/pipelines/infrastructure-service-resources.yml new file mode 100644 index 00000000..5815887b --- /dev/null +++ b/devops/pipelines/infrastructure-service-resources.yml @@ -0,0 +1,55 @@ +# Copyright © Microsoft Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +trigger: + batch: true + branches: + include: + - master + paths: + include: + - /infra/modules/providers/* + - /infra/templates/osdu-r3-mvp/service_resources/* + - /test-harness/terratest-extensions/* + exclude: + - /**/*.md + + +variables: +- name: 'TF_VERSION' + value: '0.12.29' +- name: 'GO_VERSION' + value: '1.12.14' +- name: 'TF_ROOT_DIR' + value: 'infra' +- name: 'PIPELINE_ROOT_DIR' + value: 'devops' +- name: 'TEST_HARNESS_DIR' + value: 'test-harness' +- group: 'Infrastructure Pipeline Variables' + +stages: +- template: /devops/infra-prepare-stage.yml + parameters: + environments: + - name: 'demo' + enablePrIsolation: false + resourceNameIsolationLevel: 8 + + configurations: + - jobName: service_resources + terraformWorkspacePrefix: sr + terraformTemplatePath: /infra/templates/osdu-r3-mvp/service_resources + deploymentTimeoutInMinutes: 120 diff --git a/docs/service-automation.md b/docs/service-automation.md index a183fab2..02b06fd6 100644 --- a/docs/service-automation.md +++ b/docs/service-automation.md @@ -296,14 +296,20 @@ __Create the Chart Pipelines__ Create the pipelines and run things in this exact order. 1. Add a Pipeline for __chart-osdu-common__ to deploy common components. - > Ensure DNS is configured for your Gateway IP to DNS_HOST prior. _Repo:_ `infra-azure-provisioning` - _Path:_ `/charts/osdu-common/pipeline.yml` - _Validate:_ https:// is alive. +```bash +az pipelines create \ + --name 'chart-osdu-common' \ + --repository infra-azure-provisioning \ + --branch master \ + --repository-type tfsgit \ + --yaml-path /charts/osdu-common/pipeline.yml \ + -ojson +``` 2. Add a Pipeline for __chart-osdu-istio__ to deploy Istio components. diff --git a/tools/rest/auth_token.http b/tools/rest/auth_token.http index 67f02880..6240d88e 100644 --- a/tools/rest/auth_token.http +++ b/tools/rest/auth_token.http @@ -4,7 +4,7 @@ @login_base = login.microsoftonline.com/{{TENANT_ID}} @oauth_token_host = {{login_base}}/oauth2/v2.0/token @scopes = {{CLIENT_ID}}/.default openid profile offline_access -@AUTH_CODE = +@AUTH_CODE = # ----------------------- # OAUTH authorization_code -- GitLab From 9927ee3084f4b0caf7faa97e73dc40c6c9d486fa Mon Sep 17 00:00:00 2001 From: Daniel Scholl Date: Tue, 20 Oct 2020 19:54:01 -0500 Subject: [PATCH 8/9] Updated Pipelines --- docs/infra-automation.md | 6 +- docs/service-automation.md | 111 ++++++++++++++---- infra/common_prepare.sh | 2 +- .../osdu-r3-mvp/central_resources/README.md | 2 +- .../osdu-r3-mvp/data_partition/README.md | 2 +- .../pipeline-central-resources.yml | 6 +- .../osdu-r3-mvp/pipeline-data-partition.yml | 6 +- .../pipeline-service-resources.yml | 6 +- .../osdu-r3-mvp/service_resources/README.md | 2 +- 9 files changed, 111 insertions(+), 32 deletions(-) diff --git a/docs/infra-automation.md b/docs/infra-automation.md index c8f5830d..b4328fe0 100644 --- a/docs/infra-automation.md +++ b/docs/infra-automation.md @@ -137,7 +137,7 @@ az pipelines create \ --repository infra-azure-provisioning \ --branch master \ --repository-type tfsgit \ - --yaml-path /infra/templates/osdu-r3-mvp/pipeline-central-resources.yml \ + --yaml-path /devops/pipelines/infrastructure-central-resources.yml \ -ojson ``` @@ -153,7 +153,7 @@ az pipelines create \ --repository infra-azure-provisioning \ --branch master \ --repository-type tfsgit \ - --yaml-path /infra/templates/osdu-r3-mvp/pipeline-data-partition.yml \ + --yaml-path /devops/pipelines/infrastructure-data-partition.yml \ -ojson ``` @@ -169,6 +169,6 @@ az pipelines create \ --repository infra-azure-provisioning \ --branch master \ --repository-type tfsgit \ - --yaml-path /infra/templates/osdu-r3-mvp/pipeline-service-resources.yml \ + --yaml-path /devops/pipelines/infrastructure-service-resources.yml \ -ojson ``` diff --git a/docs/service-automation.md b/docs/service-automation.md index 02b06fd6..122893a4 100644 --- a/docs/service-automation.md +++ b/docs/service-automation.md @@ -298,7 +298,7 @@ Create the pipelines and run things in this exact order. 1. Add a Pipeline for __chart-osdu-common__ to deploy common components. _Repo:_ `infra-azure-provisioning` - _Path:_ `/charts/osdu-common/pipeline.yml` + _Path:_ `/devops/pipelines/chart-osdu-common.yml` _Validate:_ https:// is alive. ```bash @@ -307,27 +307,42 @@ az pipelines create \ --repository infra-azure-provisioning \ --branch master \ --repository-type tfsgit \ - --yaml-path /charts/osdu-common/pipeline.yml \ + --yaml-path /devops/pipelines/chart-osdu-common.yml \ -ojson ``` 2. Add a Pipeline for __chart-osdu-istio__ to deploy Istio components. _Repo:_ `infra-azure-provisioning` - - _Path:_ `/charts/osdu-istio/pipeline.yml` - + _Path:_ `/devops/pipelines/chart-osdu-istio.yml` _Validate:_ Pods are running in Istio Namespace. +```bash +az pipelines create \ + --name 'chart-osdu-istio' \ + --repository infra-azure-provisioning \ + --branch master \ + --repository-type tfsgit \ + --yaml-path /devops/pipelines/chart-osdu-istio.yml \ + -ojson +``` + 3. Add a Pipeline for __chart-osdu-istio-auth__ to deploy Istio Authorization Policies. _Repo:_ `infra-azure-provisioning` - - _Path:_ `/charts/osdu-istio-auth/pipeline.yml` - + _Path:_ `/devops/pipelines/chart-osdu-istio-auth.yml` _Validate:_ Authorization Policies exist in osdu namespace. +```bash +az pipelines create \ + --name 'chart-osdu-istio-auth' \ + --repository infra-azure-provisioning \ + --branch master \ + --repository-type tfsgit \ + --yaml-path /devops/pipelines/chart-osdu-istio-auth.yml \ + -ojson +``` __Create the Service Pipelines__ @@ -336,63 +351,115 @@ Create the pipelines and run things in this exact order. 1. Add a Pipeline for __service-partition__ to deploy the Partition Service. _Repo:_ `partition` - _Path:_ `/devops/azure/pipeline.yml` - _Validate:_ https:///api/partition/v1/swagger-ui.html is alive. +```bash +az pipelines create \ + --name 'service-partition' \ + --repository partition \ + --branch master \ + --repository-type tfsgit \ + --yaml-path /devops/azure/pipeline.yml \ + -ojson +``` + 2. Add a Pipeline for __service-entitlements-azure__ to deploy the Entitlements Service. > This pipeline may have to be run twice for integration tests to pass due to a preload data issue. _Repo:_ `entitlements-azure` - _Path:_ `/devops/azure/pipeline.yml` - _Validate:_ https:///entitlements/v1/swagger-ui.html is alive. +```bash +az pipelines create \ + --name 'service-entitlements-azure' \ + --repository entitlements-azure \ + --branch master \ + --repository-type tfsgit \ + --yaml-path /devops/azure/pipeline.yml \ + -ojson +``` + 3. Add a Pipeline for __service-legal__ to deploy the Legal Service. _Repo:_ `legal` - _Path:_ `/devops/azure/pipeline.yml` - _Validate:_ https:///api/legal/v1/swagger-ui.html is alive. +```bash +az pipelines create \ + --name 'service-legal' \ + --repository legal \ + --branch master \ + --repository-type tfsgit \ + --yaml-path /devops/azure/pipeline.yml \ + -ojson +``` 4. Add a Pipeline for __service-storage__ to deploy the Storage Service. _Repo:_ `storage` - _Path:_ `/devops/azure/pipeline.yml` - _Validate:_ https:///api/storage/v2/swagger-ui.html is alive. +```bash +az pipelines create \ + --name 'service-storage' \ + --repository storage \ + --branch master \ + --repository-type tfsgit \ + --yaml-path /devops/azure/pipeline.yml \ + -ojson +``` + 5. Add a Pipeline for __service-indexer-queue__ to deploy the Indexer Queue Function. _Repo:_ `indexer-queue` - _Path:_ `/devops/azure/pipeline.yml` - _Validate:_ ScaledObject exist in osdu namespace. +```bash +az pipelines create \ + --name 'service-indexer-queue' \ + --repository indexer-queue \ + --branch master \ + --repository-type tfsgit \ + --yaml-path /devops/azure/pipeline.yml \ + -ojson +``` 6. Add a Pipeline for __service-indexer__ to deploy the Indexer Service. _Repo:_ `indexer-service` - _Path:_ `/devops/azure/pipeline.yml` - _Validate:_ https:///api/indexer/v2/swagger-ui.html is alive. +```bash +az pipelines create \ + --name 'service-indexer' \ + --repository indexer-service \ + --branch master \ + --repository-type tfsgit \ + --yaml-path /devops/azure/pipeline.yml \ + -ojson +``` 7. Add a Pipeline for __service-search__ to deploy the Search Service. _Repo:_ `search-service` - _Path:_ `/devops/azure/pipeline.yml` - _Validate:_ https:///api/search/v2/swagger-ui.html is alive. +```bash +az pipelines create \ + --name 'service-search' \ + --repository search-service \ + --branch master \ + --repository-type tfsgit \ + --yaml-path /devops/azure/pipeline.yml \ + -ojson +``` diff --git a/infra/common_prepare.sh b/infra/common_prepare.sh index 04d220e9..e0d1333d 100755 --- a/infra/common_prepare.sh +++ b/infra/common_prepare.sh @@ -546,7 +546,7 @@ export TF_VAR_remote_state_container="remote-state-container" export TF_VAR_resource_group_location="${AZURE_LOCATION}" export TF_VAR_cosmosdb_replica_location="${AZURE_PAIR_LOCATION}" -export TF_VAR_central_resources_workspace_name="${UNIQUE}-cr" +export TF_VAR_central_resources_workspace_name="cr-${UNIQUE}" export TF_VAR_principal_appId="$(az keyvault secret show --id https://$AZURE_VAULT.vault.azure.net/secrets/osdu-mvp-${UNIQUE}-principal-id --query value -otsv)" export TF_VAR_principal_name="osdu-mvp-${UNIQUE}-principal" diff --git a/infra/templates/osdu-r3-mvp/central_resources/README.md b/infra/templates/osdu-r3-mvp/central_resources/README.md index ddced8c2..91351512 100644 --- a/infra/templates/osdu-r3-mvp/central_resources/README.md +++ b/infra/templates/osdu-r3-mvp/central_resources/README.md @@ -91,7 +91,7 @@ terraform init -backend-config "storage_account_name=${TF_VAR_remote_state_accou # This command configures terraform to use a workspace unique to you. This allows you to work # without stepping over your teammate's deployments -TF_WORKSPACE="${UNIQUE}-cr" +TF_WORKSPACE="cr-${UNIQUE}" terraform workspace new $TF_WORKSPACE || terraform workspace select $TF_WORKSPACE ``` diff --git a/infra/templates/osdu-r3-mvp/data_partition/README.md b/infra/templates/osdu-r3-mvp/data_partition/README.md index 9f64acc3..bdb23ac9 100644 --- a/infra/templates/osdu-r3-mvp/data_partition/README.md +++ b/infra/templates/osdu-r3-mvp/data_partition/README.md @@ -59,7 +59,7 @@ terraform init -backend-config "storage_account_name=${TF_VAR_remote_state_accou # This command configures terraform to use a workspace unique to you. This allows you to work # without stepping over your teammate's deployments -TF_WORKSPACE="${UNIQUE}-dp" +TF_WORKSPACE="dp1-${UNIQUE}" terraform workspace new $TF_WORKSPACE || terraform workspace select $TF_WORKSPACE ``` diff --git a/infra/templates/osdu-r3-mvp/pipeline-central-resources.yml b/infra/templates/osdu-r3-mvp/pipeline-central-resources.yml index ce2108d9..a46eb88c 100644 --- a/infra/templates/osdu-r3-mvp/pipeline-central-resources.yml +++ b/infra/templates/osdu-r3-mvp/pipeline-central-resources.yml @@ -65,9 +65,13 @@ stages: - template: /devops/infra-prepare-stage.yml parameters: environments: - - name: 'demo' + - name: 'dev' enablePrIsolation: false resourceNameIsolationLevel: 8 + - ${{ if eq(variables['Build.SourceBranchName'], 'master') }}: + - name: 'glab' + enablePrIsolation: false + resourceNameIsolationLevel: 8 configurations: - jobName: central_resources diff --git a/infra/templates/osdu-r3-mvp/pipeline-data-partition.yml b/infra/templates/osdu-r3-mvp/pipeline-data-partition.yml index a0c8f177..a2012999 100644 --- a/infra/templates/osdu-r3-mvp/pipeline-data-partition.yml +++ b/infra/templates/osdu-r3-mvp/pipeline-data-partition.yml @@ -64,9 +64,13 @@ stages: - template: /devops/infra-prepare-stage.yml parameters: environments: - - name: 'demo' + - name: 'dev' enablePrIsolation: false resourceNameIsolationLevel: 8 + - ${{ if eq(variables['Build.SourceBranchName'], 'master') }}: + - name: 'glab' + enablePrIsolation: false + resourceNameIsolationLevel: 8 configurations: - jobName: data_partition_1 diff --git a/infra/templates/osdu-r3-mvp/pipeline-service-resources.yml b/infra/templates/osdu-r3-mvp/pipeline-service-resources.yml index d8463368..e34e3096 100644 --- a/infra/templates/osdu-r3-mvp/pipeline-service-resources.yml +++ b/infra/templates/osdu-r3-mvp/pipeline-service-resources.yml @@ -63,9 +63,13 @@ stages: - template: /devops/infra-prepare-stage.yml parameters: environments: - - name: 'demo' + - name: 'dev' enablePrIsolation: false resourceNameIsolationLevel: 8 + - ${{ if eq(variables['Build.SourceBranchName'], 'master') }}: + - name: 'glab' + enablePrIsolation: false + resourceNameIsolationLevel: 8 configurations: - jobName: service_resources diff --git a/infra/templates/osdu-r3-mvp/service_resources/README.md b/infra/templates/osdu-r3-mvp/service_resources/README.md index ca5b3a00..65f4826c 100644 --- a/infra/templates/osdu-r3-mvp/service_resources/README.md +++ b/infra/templates/osdu-r3-mvp/service_resources/README.md @@ -58,7 +58,7 @@ terraform init -backend-config "storage_account_name=${TF_VAR_remote_state_accou # This command configures terraform to use a workspace unique to you. This allows you to work # without stepping over your teammate's deployments -TF_WORKSPACE="${UNIQUE}-sr" +TF_WORKSPACE="sr-${UNIQUE}" terraform workspace new $TF_WORKSPACE || terraform workspace select $TF_WORKSPACE ``` -- GitLab From 89663d4a13db52fe5bdca12513f4bc5609158fdc Mon Sep 17 00:00:00 2001 From: Daniel Scholl Date: Tue, 20 Oct 2020 20:02:05 -0500 Subject: [PATCH 9/9] Added in Merge Request Templates --- .gitlab/merge_request_templates/default.md | 22 ++++++++++++++++++ .../merge_request_templates/infra_template.md | 23 +++++++++++++++++++ README.md | 7 ++++++ 3 files changed, 52 insertions(+) create mode 100644 .gitlab/merge_request_templates/default.md create mode 100644 .gitlab/merge_request_templates/infra_template.md diff --git a/.gitlab/merge_request_templates/default.md b/.gitlab/merge_request_templates/default.md new file mode 100644 index 00000000..be6a84b7 --- /dev/null +++ b/.gitlab/merge_request_templates/default.md @@ -0,0 +1,22 @@ +## All Submissions: +------------------------------------- +* [YES/NO] Have you added an explanation of what your changes do and why you'd like us to include them? +* [YES/NO] I have updated the documentation accordingly. +* [YES/NO/NA] My code follows the code style of this project. + + +## Current Behavior or Linked Issues +------------------------------------- + + + +## Does this introduce a breaking change? +------------------------------------- +- [YES/NO] + + + + +## Other information +------------------------------------- + diff --git a/.gitlab/merge_request_templates/infra_template.md b/.gitlab/merge_request_templates/infra_template.md new file mode 100644 index 00000000..e6ecde36 --- /dev/null +++ b/.gitlab/merge_request_templates/infra_template.md @@ -0,0 +1,23 @@ +## Infrastructure Submissions: +------------------------------------- +* [YES/NO] Have you added an explanation of what your changes do and why you'd like us to include them? +* [YES/NO] I have updated the documentation accordingly. +* [YES/NO/NA] I have added tests to cover my changes. +* [YES/NO/NA] All new and existing tests passed. +* [YES/NO/NA] I have formatted the terraform code. _(`terraform fmt -recursive && go fmt ./...`)_ + +## Current Behavior or Linked Issues +------------------------------------- + + + +## Does this introduce a breaking change? +------------------------------------- +- [YES/NO] + + + + +## Other information +------------------------------------- + diff --git a/README.md b/README.md index 160add6b..cdb2dc7f 100644 --- a/README.md +++ b/README.md @@ -99,6 +99,13 @@ export GIT_REPO=git@ssh.dev.azure.com:v3/${ADO_ORGANIZATION}/${ADO_PROJECT}/k8-g && git push -u origin --all) ``` +In order for Automated Pipelines to be able to work with this repository the following Permissions must be set in the ADO Project for `All Repositories/Permissions` on the user `osdu-mvp Build Service`. + +- Create Branch `Allow` +- Contribute `Allow` +- Contribute to Pull requests `Allow` + + ## Provision the Common Resources -- GitLab