infra-automation.md 6.27 KB
Newer Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
# Deploy Infrastructure

__Configure Azure DevOps Service Connection__

- Configure an [ARM Resources Service Connection](https://docs.microsoft.com/en-us/azure/devops/pipelines/library/connect-to-azure?view=azure-devops) for the desired subscription.
  - Scope should be to the desired Subscription but do not apply scope to a Resource Group

```bash
SERVICE_CONNECTION_NAME=osdu-mvp-$UNIQUE
export AZURE_DEVOPS_EXT_AZURE_RM_SERVICE_PRINCIPAL_KEY=$ARM_CLIENT_SECRET

az devops service-endpoint azurerm create \
  --name $SERVICE_CONNECTION_NAME \
  --azure-rm-tenant-id $ARM_TENANT_ID \
  --azure-rm-subscription-id $ARM_SUBSCRIPTION_ID \
  --azure-rm-subscription-name $(az account show --subscription $ARM_SUBSCRIPTION_ID --query name -otsv) \
  --azure-rm-service-principal-id $ARM_CLIENT_ID \
  -ojsonc
```


__Setup and Configure the ADO Library `Infrastructure Pipeline Variables`__

This variable group will be used to hold the common values for infrastructure to be built regardless of a specified environment.

  | Variable | Value |
  |----------|-------|
  | AGENT_POOL | Hosted Ubuntu 1604 |
  | BUILD_ARTIFACT_NAME | infra-templates |
  | SERVICE_CONNECTION_NAME | <your_service_connection_name> |
  | TF_VAR_elasticsearch_secrets_keyvault_name | osducommon<your_unique>-kv |
  | TF_VAR_elasticsearch_secrets_keyvault_resource_group | osdu-common-<your_unique> |
  | TF_VAR_remote_state_account | osducommon<your_unique> |
  | TF_VAR_remote_state_container | remote-state-container |

```bash
az pipelines variable-group create \
  --name "Infrastructure Pipeline Variables" \
  --authorize true \
  --variables \
  AGENT_POOL="Hosted Ubuntu 1604" \
  BUILD_ARTIFACT_NAME="infra-templates" \
  TF_VAR_elasticsearch_secrets_keyvault_name=$COMMON_VAULT  \
  TF_VAR_elasticsearch_secrets_keyvault_resource_group=osdu-common-${UNIQUE} \
  TF_VAR_remote_state_account=$TF_VAR_remote_state_account \
  TF_VAR_remote_state_container="remote-state-container" \
  SERVICE_CONNECTION_NAME=$SERVICE_CONNECTION_NAME \
  -ojson
```


__Setup and Configure the ADO Library `Infrastructure Pipeline Variables - demo`__

This variable group will be used to hold the common values for a specific infrastructure environment to be built. There is an implied naming convention to this Variable group `demo` relates to the environment name.  Additionally you can specify and override the region locations here.


  | Variable | Value |
  |----------|-------|
  | ARM_SUBSCRIPTION_ID | <your_subscription_id> |
  | TF_VAR_aks_agent_vm_count | 3 |
  | TF_VAR_central_resources_workspace_name | cr-demo |
  | TF_VAR_cosmosdb_replica_location | eastus2 |
  | TF_VAR_data_partition_name | opendes |
  | TF_VAR_data_resources_workspace_name | dr-demo |
  | TF_VAR_elasticsearch_version | <your_elastic_version> |
Daniel Scholl's avatar
Daniel Scholl committed
66
  | TF_VAR_gitops_branch | <desired_branch> |
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
  | TF_VAR_gitops_path | providers/azure/hld-registry |
  | TF_VAR_gitops_ssh_url | git@<your_flux_repo> |
  | TF_VAR_principal_appId | <your_principal_appId> |
  | TF_VAR_principal_name | <your_principal_name> |
  | TF_VAR_principal_objectId | <your_principal_objectId> |
  | TF_VAR_principal_password | <your_principal_password> |
  | TF_VAR_resource_group_location | centralus |

```bash
ENVIRONMENT="demo"
REGION="centralus"
REGION_PAIR="eastus2"
PARTITION_NAME="opendes"
ELASTIC_VERSION="6.8.12"
GIT_REPO=git@ssh.dev.azure.com:v3/${ADO_ORGANIZATION}/${ADO_PROJECT}/k8-gitops-manifests

az pipelines variable-group create \
  --name "Infrastructure Pipeline Variables - ${ENVIRONMENT}" \
  --authorize true \
  --variables \
  ARM_SUBSCRIPTION_ID="${ARM_SUBSCRIPTION_ID}" \
  TF_VAR_aks_agent_vm_count=3 \
  TF_VAR_central_resources_workspace_name="cr-${ENVIRONMENT}" \
  TF_VAR_cosmosdb_replica_location="${REGION_PAIR}" \
  TF_VAR_data_partition_name="${PARTITION_NAME}" \
  TF_VAR_data_resources_workspace_name="dr-${ENVIRONMENT}" \
  TF_VAR_elasticsearch_version="${ELASTIC_VERSION}" \
94
  TF_VAR_gitops_branch="${UNIQUE}" \
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
  TF_VAR_gitops_path="providers/azure/hld-registry" \
  TF_VAR_gitops_ssh_url="${GIT_REPO}" \
  TF_VAR_principal_appId="${TF_VAR_principal_appId}" \
  TF_VAR_principal_name="${TF_VAR_principal_name}" \
  TF_VAR_principal_objectId="${TF_VAR_principal_objectId}" \
  TF_VAR_principal_password="${TF_VAR_principal_password}" \
  TF_VAR_resource_group_location="${REGION}" \
  -ojson
```

__Setup and Configure the ADO Library `Infrastructure Pipeline Secrets - demo`__
> This should be linked Secrets from Azure Key Vault `osducommon<random>`

  | Variable | Value |
  |----------|-------|
  | elastic-endpoint-dp1-demo | `*********` |
  | elastic-username-dp1-demo | `*********` |
  | elastic-password-dp1-demo | `*********` |


__Setup 2 Secure Files__

[Upload the 2 Secure files](https://docs.microsoft.com/en-us/azure/devops/pipelines/library/secure-files?view=azure-devops).


  - ~/.ssh/osdu_$UNIQUE/azure-aks-gitops-ssh-key
  - ~/.ssh/osdu_$UNIQUE/azure-aks-node-ssh-key.pub



__Execute the pipelines in `osdu-infrastructure`__

> These pipelines need to be executed to completion in the specific order.

1. `infrastructure-central-resources`

  > For the first run of the pipeline approvals will need to be made for the 2 secure files and the Service Connection.

```bash
# Create and Deploy the Pipeline
az pipelines create \
  --name 'infrastructure-central-resources'  \
  --repository infra-azure-provisioning  \
  --branch master  \
  --repository-type tfsgit  \
Daniel Scholl's avatar
Daniel Scholl committed
140
  --yaml-path /devops/pipelines/infrastructure-central-resources.yml  \
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
  -ojson
```


2. `infrastructure-data-partition`

  > For the first run of the pipeline approvals will need to be made for the 2 secure files and the Service Connection.

```bash
# Create and Deploy the Pipeline
az pipelines create \
  --name 'infrastructure-data-partition'  \
  --repository infra-azure-provisioning  \
  --branch master  \
  --repository-type tfsgit  \
Daniel Scholl's avatar
Daniel Scholl committed
156
  --yaml-path /devops/pipelines/infrastructure-data-partition.yml  \
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
  -ojson
```


3. `azure-pipeline-service.yml`

  > For the first run of the pipeline approvals will need to be made for the 2 secure files and the Service Connection.

```bash
# Create and Deploy the Pipeline
az pipelines create \
  --name 'infrastructure-service-resources'  \
  --repository infra-azure-provisioning  \
  --branch master  \
  --repository-type tfsgit  \
Daniel Scholl's avatar
Daniel Scholl committed
172
  --yaml-path /devops/pipelines/infrastructure-service-resources.yml  \
173
174
  -ojson
```