Added in additional items for Service Pipelines

README.md

@@ -19,7 +19,7 @@ Azure environment cost ballpark [estimate](https://tinyurl.com/y4e9s7rf). This i
 1. Azure Subscription
 1. Terraform and Go are locally installed.
 1. Requires the use of [direnv](https://direnv.net/).
-1. Install the required common tools (kubectl, helm, and terraform).  
+1. Install the required common tools (kubectl, helm, and terraform).
 
 
 ### Install the required tooling
@@ -114,6 +114,35 @@ export UNIQUE=demo
 ./infra/common_prepare.sh $(az account show --query id -otsv) $UNIQUE
 ```
 
+Integration Tests requires 2 Azure AD User Accounts, a tenant user and a guest user to be setup in order to use for integration testing.  This activity needs to be performed by someone who has access as an AD User Admin.
+
+- ad-guest-email (ie: integration.test@email.com)
+- ad-guest-oid (OID of the user)
+- ad-user-email (ie: integration.test@{tenant}.onmicrosoft.com
+- ad-user-oid (OID of the user)
+
+```bash
+USER_EMAIL=""
+USER_EMAIL_OID=""
+GUEST_EMAIL=""
+GUEST_EMAIL_OID=""
+
+az keyvault secret set --vault-name $COMMON_VAULT --name "ad-user-email" --value $USER_EMAIL
+az keyvault secret set --vault-name $COMMON_VAULT --name "ad-user-oid" --value $USER_EMAIL_OID
+az keyvault secret set --vault-name $COMMON_VAULT --name "ad-guest-email" --value $GUEST_EMAIL
+az keyvault secret set --vault-name $COMMON_VAULT --name "ad-guest-oid" --value $GUEST_EMAIL_OID
+```
+
+Istio Configuration setups a Dashboard that requires some admin credentials.  Automation Pipelines uses settings out of the common keyvault for applying the values of the Istio Dashboard default credentials.
+
+```bash
+ISTIO_USERNAME=""
+ISTIO_PASSWORD=""
+
+
+az keyvault secret set --vault-name $COMMON_VAULT --name "istio-username" --value $(echo $ISTIO_USERNAME |base64)
+az keyvault secret set --vault-name $COMMON_VAULT --name "istio-password" --value $(echo $ISTIO_PASSWORD |base64)
+```
 
 __Local Script Output Resources__
 
@@ -193,27 +222,43 @@ az keyvault secret show \
 ```
 
 
-## Automated Pipeline Installation
+## Install OSDU
 
-> This typically takes about 3 hours to complete.
+There are 2 methods that can be chosen to perform installation at this point in time.
 
-1. Configure the Pipelines following directions [here](./docs/pipeline-setup.md).
+1. Manual Installation -- Typically used when the desire is to manually make modifications to the environment and have full control all updates and deployments.
 
-2. Manually configure your DNS_HOST to the IP Address of the environment IP Address.
+2. Pipeline Installation -- Typically used when the need is to only access the Data Platform but allow for automatic upgrades of infrastructure and services.
 
-3. Deploy the application helm charts following the directions [here]().
 
+__Manual Installation__
 
+> This typically takes about 2 hours to complete.
 
-## Manual Installation
+1. Install the Infrastructure following directions [here](./infra/templates/osdu-r3-mvp).
 
-> This typically takes about 2 hours to complete.
+2. Setup DNS to point to the deployed infrastructure following directions [here](./docs/dns-setup.md).
+
+3. Upload the Integration Test Data following directions [here](./tools/test_data).
+
+3. Deploy the application helm charts following the directions [here](./charts).
+
+5. Setup Environment Variables for IDE Development and Integration Testing.
+
+
+__Automated Pipeline Installation__
+
+> This typically takes about 3 hours to complete.
+
+1. Setup Code Mirroring following directions [here](./docs/code-mirroring.md).
+
+2. Setup Infrastructure Automation following directions [here](./docs/infra-automation.md).
 
-1. Install the Infrastructure following directions [here](./infra/templates/osdu-r3-mvp/README.md).
+2. Setup DNS to point to the deployed infrastructure following directions [here](./docs/dns-setup.md).
 
-2. Manually configure your DNS_HOST to the IP Address of the environment IP Address.
+4. Upload the Integration Test Data following directions [here](./tools/test_data).
 
-3. Deploy the application helm charts following the directions [here](./charts/README.md).
+5. Setup Service Automation following directions [here](./docs/service-automation.md).
 
 
 

docs/code-mirroring.md

+# Setup Mirroring for Gitlab Repositories
+
+__Create Empty Repositories__
+
+Empty repositories need to be created that will be used by a pipeline to mirror gitlab repositories into.
+
+| Repository Name           | Gitlab Location
+|---------------------------|---------------------------|
+| infra-azure-provisioning  | https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning.git |
+| partition                 | https://community.opengroup.org/osdu/platform/system/partition.git |
+| entitlements-azure        | https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements-azure.git |
+| legal                     | https://community.opengroup.org/osdu/platform/security-and-compliance/legal.git |
+| indexer-queue             | https://community.opengroup.org/osdu/platform/system/indexer-queue.git |
+| storage                   | https://community.opengroup.org/osdu/platform/system/storage.git |
+| indexer-service           | https://community.opengroup.org/osdu/platform/system/indexer-service.git |
+| search-service            | https://community.opengroup.org/osdu/platform/system/search-service.git |
+| delivery                  | https://community.opengroup.org/osdu/platform/system/delivery.git       |
+
+```bash
+export ADO_ORGANIZATION=<organization_name>
+export ADO_PROJECT=osdu-mvp
+
+az devops configure --defaults organization=https://dev.azure.com/$ADO_ORGANIZATION project=$ADO_PROJECT
+
+# Create required ADO Repositories
+for SERVICE in infra-azure-provisioning partition entitlements-azure legal storage indexer-queue indexer-service search-service delivery;
+do
+  az repos create --name $SERVICE --organization https://dev.azure.com/${ADO_ORGANIZATION} --project $ADO_PROJECT -ojson
+done
+```
+
+
+__Create Variable Group__
+
+This variable group will be used to hold the values of the GitLab Location to be mirrored.  Additionally a Personal Access Token is necessary to allow for git checkin.
+
+Variable Group Name:  `Mirror Variables`
+
+| Variable | Value |
+|----------|-------|
+| OSDU_INFRASTRUCTURE | https://dev.azure.com/osdu-demo/osdu/_git/osdu-infrastructure |
+| INFRA_PROVISIONING_REPO | https://dev.azure.com/osdu-demo/osdu/_git/infra-azure-provisioning |
+| PARTITION_REPO | https://dev.azure.com/osdu-demo/osdu/_git/partition |
+| ENTITLEMENTS_REPO | https://dev.azure.com/osdu-demo/osdu/_git/entitlements-azure |
+| LEGAL_REPO | https://dev.azure.com/osdu-demo/osdu/_git/legal |
+| STORAGE_REPO | https://dev.azure.com/osdu-demo/osdu/_git/storage |
+| INDEXER_QUEUE_REPO | https://dev.azure.com/osdu-demo/osdu/_git/indexer-queue |
+| INDEXER_REPO | https://dev.azure.com/osdu-demo/osdu/_git/indexer-service |
+| SEARCH_REPO | https://dev.azure.com/osdu-demo/osdu/_git/search-service |
+| DELIVERY_REPO | https://dev.azure.com/osdu-demo/osdu/_git/delivery |
+| ACCESS_TOKEN | <your_personal_access_token> |
+
+
+Manually create a Personal Access Token following the [documentation](https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&tabs=preview-page) and add a Variable called `ACCESS_TOKEN` with the value being the PAT created.
+
+
+```bash
+ACCESS_TOKEN=<your_access_token>
+
+az pipelines variable-group create \
+  --name "Mirror Variables" \
+  --authorize true \
+  --variables \
+  INFRA_PROVISIONING_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/infra-azure-provisioning \
+  PARTITION_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/partition \
+  ENTITLEMENTS_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/entitlements-azure \
+  LEGAL_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/legal \
+  STORAGE_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/storage \
+  INDEXER_QUEUE_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/indexer-queue \
+  INDEXER_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/indexer-service \
+  SEARCH_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/search-service \
+  DELIVERY_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/delivery \
+  ACCESS_TOKEN=$ACCESS_TOKEN \
+  -ojson
+```
+
+
+__Create Mirror Pipeline__
+
+Clone the Project Repository `osdu-mvp`, and add the pipeline.
+
+
+```bash
+GIT_SSH_COMMAND="ssh -i ${TF_VAR_gitops_ssh_key_file}"  \
+  git clone git@ssh.dev.azure.com:v3/${ADO_ORGANIZATION}/${ADO_PROJECT}/${ADO_PROJECT}
+
+cat > ${ADO_PROJECT}/pipeline.yml << 'EOF'
+#  Copyright © Microsoft Corporation
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+
+### UNCOMMENT IF YOU WANT A SCHEDULED PULL ####
+
+# schedules:
+#   - cron: "*/10 * * * *"
+#     displayName: Hourly Pull Schedule
+#     branches:
+#       include:
+#       - master
+#     always: true
+
+variables:
+  - group: 'Mirror Variables'
+
+jobs:
+  - job: mirror_sync
+    displayName: 'Pull Repositories'
+    steps:
+
+    - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1
+      displayName: 'infra-azure-provisioning'
+      inputs:
+        sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning.git'
+        destinationGitRepositoryUri: '$(INFRA_PROVISIONING_REPO)'
+        destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN)
+
+    - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1
+      displayName: 'partition'
+      inputs:
+        sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/system/partition.git'
+        destinationGitRepositoryUri: '$(PARTITION_REPO)'
+        destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN)
+
+    - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1
+      displayName: 'entitlements-azure'
+      inputs:
+        sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements-azure.git'
+        destinationGitRepositoryUri: '$(ENTITLEMENTS_REPO)'
+        destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN)
+
+    - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1
+      displayName: 'legal'
+      inputs:
+        sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/security-and-compliance/legal.git'
+        destinationGitRepositoryUri: '$(LEGAL_REPO)'
+        destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN)
+
+    - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1
+      displayName: 'indexer-queue'
+      inputs:
+        sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/system/indexer-queue.git'
+        destinationGitRepositoryUri: '$(INDEXER_QUEUE_REPO)'
+        destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN)
+
+    - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1
+      displayName: 'storage'
+      inputs:
+        sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/system/storage.git'
+        destinationGitRepositoryUri: '$(STORAGE_REPO)'
+        destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN)
+
+    - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1
+      displayName: 'indexer-service'
+      inputs:
+        sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/system/indexer-service.git'
+        destinationGitRepositoryUri: '$(INDEXER_REPO)'
+        destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN)
+
+    - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1
+      displayName: 'search-service'
+      inputs:
+        sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/system/search-service.git'
+        destinationGitRepositoryUri: '$(SEARCH_REPO)'
+        destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN)
+
+    - task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1
+      displayName: 'delivery'
+      inputs:
+        sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/system/delivery.git'
+        destinationGitRepositoryUri: '$(DELIVERY_REPO)'
+        destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN)
+EOF
+
+(cd ${ADO_PROJECT}  && git add -A && git commit -m "pipeline" && git push)
+rm -rf ${ADO_PROJECT}
+
+# Create and Execute the Pipeline
+az pipelines create \
+  --name 'gitlab-sync'  \
+  --repository $ADO_PROJECT  \
+  --branch master  \
+  --repository-type tfsgit  \
+  --yaml-path /pipeline.yml  \
+  -ojson
+```
+

docs/dns-setup.md

+# DNS Setup
+
+Manually update your DNS A Records to point to the Public IP Address for the environment.
+
+```bash
+# Get IP Address
+RESOURCE_GROUP=$(az group list --query "[?contains(name, '${UNIQUE}sr')].name" -otsv |grep -v MC)
+az network public-ip list --resource-group $RESOURCE_GROUP --query [].ipAddress -otsv
+```

docs/infra-automation.md

+# Deploy Infrastructure
+
+__Configure Azure DevOps Service Connection__
+
+- Configure an [ARM Resources Service Connection](https://docs.microsoft.com/en-us/azure/devops/pipelines/library/connect-to-azure?view=azure-devops) for the desired subscription.
+  - Scope should be to the desired Subscription but do not apply scope to a Resource Group
+
+```bash
+SERVICE_CONNECTION_NAME=osdu-mvp-$UNIQUE
+export AZURE_DEVOPS_EXT_AZURE_RM_SERVICE_PRINCIPAL_KEY=$ARM_CLIENT_SECRET
+
+az devops service-endpoint azurerm create \
+  --name $SERVICE_CONNECTION_NAME \
+  --azure-rm-tenant-id $ARM_TENANT_ID \
+  --azure-rm-subscription-id $ARM_SUBSCRIPTION_ID \
+  --azure-rm-subscription-name $(az account show --subscription $ARM_SUBSCRIPTION_ID --query name -otsv) \
+  --azure-rm-service-principal-id $ARM_CLIENT_ID \
+  -ojsonc
+```
+
+
+__Setup and Configure the ADO Library `Infrastructure Pipeline Variables`__
+
+This variable group will be used to hold the common values for infrastructure to be built regardless of a specified environment.
+
+  | Variable | Value |
+  |----------|-------|
+  | AGENT_POOL | Hosted Ubuntu 1604 |
+  | BUILD_ARTIFACT_NAME | infra-templates |
+  | SERVICE_CONNECTION_NAME | <your_service_connection_name> |
+  | TF_VAR_elasticsearch_secrets_keyvault_name | osducommon<your_unique>-kv |
+  | TF_VAR_elasticsearch_secrets_keyvault_resource_group | osdu-common-<your_unique> |
+  | TF_VAR_remote_state_account | osducommon<your_unique> |
+  | TF_VAR_remote_state_container | remote-state-container |
+
+```bash
+az pipelines variable-group create \
+  --name "Infrastructure Pipeline Variables" \
+  --authorize true \
+  --variables \
+  AGENT_POOL="Hosted Ubuntu 1604" \
+  BUILD_ARTIFACT_NAME="infra-templates" \
+  TF_VAR_elasticsearch_secrets_keyvault_name=$COMMON_VAULT  \
+  TF_VAR_elasticsearch_secrets_keyvault_resource_group=osdu-common-${UNIQUE} \
+  TF_VAR_remote_state_account=$TF_VAR_remote_state_account \
+  TF_VAR_remote_state_container="remote-state-container" \
+  SERVICE_CONNECTION_NAME=$SERVICE_CONNECTION_NAME \
+  -ojson
+```
+
+
+__Setup and Configure the ADO Library `Infrastructure Pipeline Variables - demo`__
+
+This variable group will be used to hold the common values for a specific infrastructure environment to be built. There is an implied naming convention to this Variable group `demo` relates to the environment name.  Additionally you can specify and override the region locations here.
+
+
+  | Variable | Value |
+  |----------|-------|
+  | ARM_SUBSCRIPTION_ID | <your_subscription_id> |
+  | TF_VAR_aks_agent_vm_count | 3 |
+  | TF_VAR_central_resources_workspace_name | cr-demo |
+  | TF_VAR_cosmosdb_replica_location | eastus2 |
+  | TF_VAR_data_partition_name | opendes |
+  | TF_VAR_data_resources_workspace_name | dr-demo |
+  | TF_VAR_elasticsearch_version | <your_elastic_version> |
+  | TF_VAR_gitops_branch | master |
+  | TF_VAR_gitops_path | providers/azure/hld-registry |
+  | TF_VAR_gitops_ssh_url | git@<your_flux_repo> |
+  | TF_VAR_principal_appId | <your_principal_appId> |
+  | TF_VAR_principal_name | <your_principal_name> |
+  | TF_VAR_principal_objectId | <your_principal_objectId> |
+  | TF_VAR_principal_password | <your_principal_password> |
+  | TF_VAR_resource_group_location | centralus |
+
+```bash
+ENVIRONMENT="demo"
+REGION="centralus"
+REGION_PAIR="eastus2"
+PARTITION_NAME="opendes"
+ELASTIC_VERSION="6.8.12"
+GIT_REPO=git@ssh.dev.azure.com:v3/${ADO_ORGANIZATION}/${ADO_PROJECT}/k8-gitops-manifests
+
+az pipelines variable-group create \
+  --name "Infrastructure Pipeline Variables - ${ENVIRONMENT}" \
+  --authorize true \
+  --variables \
+  ARM_SUBSCRIPTION_ID="${ARM_SUBSCRIPTION_ID}" \
+  TF_VAR_aks_agent_vm_count=3 \
+  TF_VAR_central_resources_workspace_name="cr-${ENVIRONMENT}" \
+  TF_VAR_cosmosdb_replica_location="${REGION_PAIR}" \
+  TF_VAR_data_partition_name="${PARTITION_NAME}" \
+  TF_VAR_data_resources_workspace_name="dr-${ENVIRONMENT}" \
+  TF_VAR_elasticsearch_version="${ELASTIC_VERSION}" \
+  TF_VAR_gitops_branch="master" \
+  TF_VAR_gitops_path="providers/azure/hld-registry" \
+  TF_VAR_gitops_ssh_url="${GIT_REPO}" \
+  TF_VAR_principal_appId="${TF_VAR_principal_appId}" \
+  TF_VAR_principal_name="${TF_VAR_principal_name}" \
+  TF_VAR_principal_objectId="${TF_VAR_principal_objectId}" \
+  TF_VAR_principal_password="${TF_VAR_principal_password}" \
+  TF_VAR_resource_group_location="${REGION}" \
+  -ojson
+```
+
+__Setup and Configure the ADO Library `Infrastructure Pipeline Secrets - demo`__
+> This should be linked Secrets from Azure Key Vault `osducommon<random>`
+
+  | Variable | Value |
+  |----------|-------|
+  | elastic-endpoint-dp1-demo | `*********` |
+  | elastic-username-dp1-demo | `*********` |
+  | elastic-password-dp1-demo | `*********` |
+
+
+__Setup 2 Secure Files__
+
+[Upload the 2 Secure files](https://docs.microsoft.com/en-us/azure/devops/pipelines/library/secure-files?view=azure-devops).
+
+
+  - ~/.ssh/osdu_$UNIQUE/azure-aks-gitops-ssh-key
+  - ~/.ssh/osdu_$UNIQUE/azure-aks-node-ssh-key.pub
+
+
+
+__Execute the pipelines in `osdu-infrastructure`__
+
+> These pipelines need to be executed to completion in the specific order.
+
+1. `infrastructure-central-resources`
+
+  > For the first run of the pipeline approvals will need to be made for the 2 secure files and the Service Connection.
+
+```bash
+# Create and Deploy the Pipeline
+az pipelines create \
+  --name 'infrastructure-central-resources'  \
+  --repository infra-azure-provisioning  \
+  --branch master  \
+  --repository-type tfsgit  \
+  --yaml-path /infra/templates/osdu-r3-mvp/pipeline-central-resources.yml  \
+  -ojson
+```
+
+
+2. `infrastructure-data-partition`
+
+  > For the first run of the pipeline approvals will need to be made for the 2 secure files and the Service Connection.
+
+```bash
+# Create and Deploy the Pipeline
+az pipelines create \
+  --name 'infrastructure-data-partition'  \
+  --repository infra-azure-provisioning  \
+  --branch master  \
+  --repository-type tfsgit  \
+  --yaml-path /infra/templates/osdu-r3-mvp/pipeline-data-partition.yml  \
+  -ojson
+```
+
+
+3. `azure-pipeline-service.yml`
+
+  > For the first run of the pipeline approvals will need to be made for the 2 secure files and the Service Connection.
+
+```bash
+# Create and Deploy the Pipeline
+az pipelines create \
+  --name 'infrastructure-service-resources'  \
+  --repository infra-azure-provisioning  \
+  --branch master  \
+  --repository-type tfsgit  \
+  --yaml-path /infra/templates/osdu-r3-mvp/pipeline-service-resources.yml  \
+  -ojson
+```

infra/README.md

@@ -60,18 +60,3 @@ $ tree -d
                 ├── integration
                 └── unit
 ```
-
-## License
-Copyright © Microsoft Corporation
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.

infra/common_prepare.sh

@@ -211,9 +211,14 @@ function CreateADApplication() {
         --display-name $1 \
         --query [].appId -otsv)
 
+      APP_OID=$(az ad app list \
+        --display-name $1 \
+        --query [].objectId -otsv)
+
       tput setaf 2; echo "Adding AD Application to Vault..." ; tput sgr0
       AddKeyToVault $2 "${1}-clientid" $APP_ID
       AddKeyToVault $2 "${1}-secret" $APP_SECRET
+      AddKeyToVault $2 "${1}-oid" $APP_OID
 
     else
         tput setaf 3;  echo "AD Application $1 already exists."; tput sgr0

infra/templates/osdu-r3-mvp/README.md

@@ -15,19 +15,3 @@ Follow the directions in the [`data_resources`](./data_partition/README.md) envi
 __Deploy Service Resources__
 
 Follow the directions in the [`service_resources`](./service_resources/README.md) environment.
-
-## License
-
-Copyright © Microsoft Corporation
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.

infra/templates/osdu-r3-mvp/central_resources/README.md

@@ -127,20 +127,3 @@ Integration tests can be run using the following command:
 ```
 go test -v $(go list ./... | grep "integration")
 ```
-
-
-## License
-
-Copyright © Microsoft Corporation
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.

infra/templates/osdu-r3-mvp/data_partition/README.md

@@ -95,20 +95,3 @@ Integration tests can be run using the following command:
 ```
 go test -v $(go list ./... | grep "integration")
 ```
-
-
-## License
-
-Copyright © Microsoft Corporation
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.

infra/templates/osdu-r3-mvp/service_resources/README.md

@@ -94,20 +94,3 @@ Integration tests can be run using the following command:
 ```
 go test -v $(go list ./... | grep "integration")
 ```
-
-
-## License
-
-Copyright © Microsoft Corporation
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.

tools/test_data/Legal_COO.json

+[
+  {
+    "name": "Malaysia",
+    "alpha2": "MY",
+    "numeric": 458,
+    "residencyRisk": "Client consent required"
+  }
+]

tools/test_data/README.md

+# Test Data Upload Instructions
+
+## Getting started
+
+* [Python 2.7 or 3.5.3+][python]
+
+__SDK installation__
+
+Install the Python SDK
+