Skip to content
GitLab
Commit 9a42fe59 authored by Daniel Scholl's avatar Daniel Scholl
Browse files

Updated Scripts and Fixed bugs

parent f71d5be9
Hide whitespace changes
Inline Side-by-side
README.md
View file @ 9a42fe59
......@@ -70,14 +70,13 @@ The script `common_prepare.sh` script is a _helper_ script designed to help setu
- Ensure you are logged into the azure cli with the desired subscription set.
- Ensure you have the access to run az ad commands.
```bash
# Login to Azure CLI and ensure subscription is set to desired subscription
az login
az account set --subscription <your_subscription>
# Execute Script
UNIQUE=demo # 3-8 characters
UNIQUE=demo
./infra/templates/osdu-r3-mvp/common_prepare.sh $(az account show --query id -otsv) $UNIQUE
```
......@@ -95,12 +94,12 @@ The script creates some local files to be used.
1. .envrc_{UNIQUE} -- This is a copy of the required environment variables for the common components.
2. .envrc -- This file is used directory by direnv and requires `direnv allow` to be run to access variables.
3. .ssh/azure-aks-gitops-ssh-key -- SSH key used by flux.
4. .ssh/azure-aks-gitops-key.pub -- SSH Public Key used by flux.
5. .ssh/azure-aks-gitops-key.passphrase -- SSH Key Passphrase used by flux.
6. .ssh/azure-aks-node-ssh-key -- SSH Key used by AKS
7. .ssh/azure-aks-node-ssh-key.pub -- SSH Public Key used by AKS
8. .ssh/azure-aks-node-ssh-key.passphrase -- SSH Key Passphrase used by AKS
3. ~/.ssh/osdu_{UNIQUE}/azure-aks-gitops-ssh-key -- SSH key used by flux.
4. ~/.ssh/osdu_{UNIQUE}/azure-aks-gitops-key.pub -- SSH Public Key used by flux.
5. ~/.ssh/osdu_{UNIQUE}/azure-aks-gitops-key.passphrase -- SSH Key Passphrase used by flux.
6. ~/.ssh/osdu_{UNIQUE}/azure-aks-node-ssh-key -- SSH Key used by AKS
7. ~/.ssh/osdu_{UNIQUE}/azure-aks-node-ssh-key.pub -- SSH Public Key used by AKS
8. ~/.ssh/osdu_{UNIQUE}/azure-aks-node-ssh-key.passphrase -- SSH Key Passphrase used by AKS
__Installed Common Resources__
......@@ -129,27 +128,32 @@ az keyvault secret set --vault-name $COMMON_VAULT --name "elastic-endpoint-dp1-d
az keyvault secret set --vault-name $COMMON_VAULT --name "elastic-username-dp1-demo" --value $USERNAME
az keyvault secret set --vault-name $COMMON_VAULT --name "elastic-password-dp1-demo" --value $PASSWORD
cat >> .envrc_${UNIQUE} << EOF
cat >> .envrc << EOF
# https://cloud.elastic.co
# ------------------------------------------------------------------------------------------------------
export TF_VAR_elasticsearch_endpoint="$(az keyvault secret show --vault-name $COMMON_VAULT --id https://$COMMON_VAULT.vault.azure.net/secrets/elastic-endpoint-ado-demo --query value -otsv)"
export TF_VAR_elasticsearch_username="$(az keyvault secret show --vault-name $COMMON_VAULT --id https://$COMMON_VAULT.vault.azure.net/secrets/elastic-username-ado-demo --query value -otsv)"
export TF_VAR_elasticsearch_password="$(az keyvault secret show --vault-name $COMMON_VAULT --id https://$COMMON_VAULT.vault.azure.net/secrets/elastic-password-ado-demo --query value -otsv)"
export TF_VAR_elasticsearch_endpoint="$(az keyvault secret show --vault-name $COMMON_VAULT --id https://$COMMON_VAULT.vault.azure.net/secrets/elastic-endpoint-dp1-demo --query value -otsv)"
export TF_VAR_elasticsearch_username="$(az keyvault secret show --vault-name $COMMON_VAULT --id https://$COMMON_VAULT.vault.azure.net/secrets/elastic-username-dp1-demo --query value -otsv)"
export TF_VAR_elasticsearch_password="$(az keyvault secret show --vault-name $COMMON_VAULT --id https://$COMMON_VAULT.vault.azure.net/secrets/elastic-password-dp1-demo --query value -otsv)"
EOF
cp .envrc_${UNIQUE} .envrc
cp .envrc .envrc_${UNIQUE}
```
## Configure Key Access in Manifest Repository
The public key of the [RSA key pair](#create-an-rsa-key-pair-for-a-deploy-key-for-the-flux-repository) previously created needs to be added as a deploy key. Note: _If you do not own the repository, you will have to fork it before proceeding_.
Use the contents of the Secret as shown above.
The public key of the `azure-aks-gitops-ssh-key` previously created needs to be added as a deploy key in your Azure DevOPS Project, follow these [steps](https://docs.microsoft.com/en-us/azure/devops/repos/git/use-ssh-keys-to-authenticate?view=azure-devops&tabs=current-page#step-2--add-the-public-key-to-azure-devops-servicestfs) to add your public SSH key to your ADO environment.
Next, in your Azure DevOPS Project, follow these [steps](https://docs.microsoft.com/en-us/azure/devops/repos/git/use-ssh-keys-to-authenticate?view=azure-devops&tabs=current-page#step-2--add-the-public-key-to-azure-devops-servicestfs) to add your public SSH key to your ADO environment.
```bash
# Retrieve the public key
az keyvault secret show \
--vault-name $COMMON_VAULT \
--id https://$COMMON_VAULT.vault.azure.net/secrets/azure-aks-gitops-ssh-key-pub \
--query value \
-otsv
```
## Automated Pipeline Installation
......
docs/pipeline-setup.md
View file @ 9a42fe59
......@@ -3,7 +3,7 @@
## Setup Mirroring of Gitlab Repositories
> This typically takes about 30 minutes to complete.
> This typically takes about 10 minutes to complete.
__Create Empty Repositories__
......@@ -54,7 +54,12 @@ Variable Group Name: `Mirror Variables`
| ACCESS_TOKEN | <your_personal_access_token> |
Manually create a Personal Access Token following the [documentation](https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&tabs=preview-page) and add a Variable called `ACCESS_TOKEN` with the value being the PAT created.
```bash
ACCESS_TOKEN=<your_access_token>
az pipelines variable-group create \
--name "Mirror Variables" \
--authorize true \
......@@ -70,17 +75,21 @@ az pipelines variable-group create \
INDEXER_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/indexer-service \
SEARCH_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/search-service \
DELIVERY_REPO=https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/delivery \
ACCESS_TOKEN=$ACCESS_TOKEN \
-ojson
```
Manually create a Personal Access Token following the [documentation](https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&tabs=preview-page) and add a Variable called `ACCESS_TOKEN` with the value being the PAT created.
__Create Mirror Pipeline__
Clone the Project Repository `osdu-mvp`, and add the pipeline.
__Create Pipeline__
Manually Create a Pipeline [`gitlab-sync`](../devops/gitlab-sync.yml)
```bash
GIT_SSH_COMMAND="ssh -i ${TF_VAR_gitops_ssh_key_file}" \
git clone git@ssh.dev.azure.com:v3/${ADO_ORGANIZATION}/${ADO_PROJECT}/${ADO_PROJECT}
```yaml
cat > ${ADO_PROJECT}/pipeline.yml << 'EOF'
# Copyright © Microsoft Corporation
#
# Licensed under the Apache License, Version 2.0 (the "License");
......@@ -176,12 +185,21 @@ jobs:
sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/system/delivery.git'
destinationGitRepositoryUri: '$(DELIVERY_REPO)'
destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN)
```
EOF
Execute the Pipeline which will then `git clone --mirror` the repositories into ADO.
(cd ${ADO_PROJECT} && git add -A && git commit -m "pipeline" && git push)
rm -rf ${ADO_PROJECT}
```bash
az pipelines run --name gitlab-sync --organization https://dev.azure.com/${ADO_ORGANIZATION} --project $ADO_PROJECT -ojson
# Create and Execute the Pipeline
az pipelines create \
--name 'gitlab-sync' \
--repository $ADO_PROJECT \
--branch master \
--repository-type tfsgit \
--yaml-path /pipeline.yml \
--organization https://dev.azure.com/${ADO_ORGANIZATION} \
--project $ADO_PROJECT \
-ojson
```
......@@ -195,7 +213,7 @@ __Configure Azure DevOps Service Connection__
- Scope should be to the desired Subscription but do not apply scope to a Resource Group
```bash
SERVICE_CONNECTION_NAME=osdu-mvp-connection
SERVICE_CONNECTION_NAME=osdu-mvp-$UNIQUE
export AZURE_DEVOPS_EXT_AZURE_RM_SERVICE_PRINCIPAL_KEY=$ARM_CLIENT_SECRET
az devops service-endpoint azurerm create \
......@@ -223,8 +241,6 @@ __Setup and Configure the ADO Library `Infrastructure Pipeline Variables`__
| TF_VAR_remote_state_container | remote-state-container |
```bash
SERVICE_CONNECTION_NAME=osdu-mvp-connection
az pipelines variable-group create \
--name "Infrastructure Pipeline Variables" \
--authorize true \
......@@ -298,7 +314,7 @@ az pipelines variable-group create \
```
__Setup and Configure the ADO Library `Infrastructure Pipeline Secrets - demo`__
> This should be linked Secrets from Azure Key Vault `osducommon<your_unique>-kv`
> This should be linked Secrets from Azure Key Vault `osducommon<random>`
| Variable | Value |
|----------|-------|
......@@ -312,8 +328,8 @@ __Setup 2 Secure Files__
[Upload the 2 Secure files](https://docs.microsoft.com/en-us/azure/devops/pipelines/library/secure-files?view=azure-devops).
- azure-aks-gitops-ssh-key
- azure-aks-node-ssh-key.pub
- ~/.ssh/osdu_$UNIQUE/azure-aks-gitops-ssh-key
- ~/.ssh/osdu_$UNIQUE/azure-aks-node-ssh-key.pub
......
infra/templates/osdu-r3-mvp/common_prepare.sh
View file @ 9a42fe59
......@@ -133,7 +133,7 @@ function CreateTfPrincipal() {
tput setaf 2; echo "Adding Access Policy..." ; tput sgr0
az keyvault set-policy --name $AZURE_VAULT \
--object-id $(az ad app list --display-name $1 --query [].objectId -otsv) \
--object-id $(az ad sp list --display-name $1 --query [].objectId -otsv) \
--secret-permissions list get \
-ojson
......@@ -524,4 +524,5 @@ export TF_VAR_gitops_ssh_url="${GIT_REPO}"
export TF_VAR_gitops_branch="${UNIQUE}"
EOF
cp .envrc .envrc_${UNIQUE}
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment