README.md 13.1 KB
Newer Older
ethiraj krishnamanaidu's avatar
ethiraj krishnamanaidu committed
1
2
# infra-azure-provisioning

Daniel Scholl's avatar
Daniel Scholl committed
3
This repository contains the infrastructure as code implementation and pipelines necessary for the required infrastructure to host OSDU on Azure.
Dania Kodeih (Microsoft)'s avatar
Dania Kodeih (Microsoft) committed
4

Daniel Scholl's avatar
Daniel Scholl committed
5
The `osdu` - R3 MVP Architecture solution template is intended to provision Managed Kubernetes resources like AKS and other core OSDU cloud managed services like Cosmos, Blob Storage and Keyvault.
Dania Kodeih (Microsoft)'s avatar
Dania Kodeih (Microsoft) committed
6

Daniel Scholl's avatar
Daniel Scholl committed
7
## Cloud Resource Architecture
Dania Kodeih (Microsoft)'s avatar
Dania Kodeih (Microsoft) committed
8

Daniel Scholl's avatar
Daniel Scholl committed
9
![Architecture](./docs/images/architecture.png "Architecture")
Dania Kodeih (Microsoft)'s avatar
Dania Kodeih (Microsoft) committed
10

Dania Kodeih (Microsoft)'s avatar
Dania Kodeih (Microsoft) committed
11

Daniel Scholl's avatar
Daniel Scholl committed
12
## Cost
13

Daniel Scholl's avatar
Daniel Scholl committed
14
Azure environment cost ballpark [estimate](https://tinyurl.com/y4e9s7rf). This is subject to change and is driven from the resource pricing tiers configured when the template is deployed.
15

Daniel Scholl's avatar
Daniel Scholl committed
16

Daniel Scholl's avatar
Daniel Scholl committed
17
## Prerequisites
Daniel Scholl's avatar
Daniel Scholl committed
18

Daniel Scholl's avatar
Daniel Scholl committed
19
20
21
1. Azure Subscription
1. Terraform and Go are locally installed.
1. Requires the use of [direnv](https://direnv.net/).
22
1. Install the required common tools (kubectl, helm, and terraform).
Daniel Scholl's avatar
Daniel Scholl committed
23
24


Daniel Scholl's avatar
Daniel Scholl committed
25
### Install the required tooling
Daniel Scholl's avatar
Daniel Scholl committed
26

Daniel Scholl's avatar
Daniel Scholl committed
27
This document assumes one is running a current version of Ubuntu. Windows users can install the Ubuntu Terminal from the Microsoft Store. The Ubuntu Terminal enables Linux command-line utilities, including bash, ssh, and git that will be useful for the following deployment. _Note: You will need the Windows Subsystem for Linux installed to use the Ubuntu Terminal on Windows_.
Daniel Scholl's avatar
Daniel Scholl committed
28

Daniel Scholl's avatar
Daniel Scholl committed
29

30
Currently the versions in use are [Terraform 0.14.4](https://releases.hashicorp.com/terraform/0.14.4/) and [GO 1.12.14](https://golang.org/dl/).
Daniel Scholl's avatar
Daniel Scholl committed
31

Daniel Scholl's avatar
Daniel Scholl committed
32
> Note: Terraform and Go are recommended to be installed using a [Terraform Version Manager](https://github.com/tfutils/tfenv) and a [Go Version Manager](https://github.com/stefanmaric/g)
Daniel Scholl's avatar
Daniel Scholl committed
33
34


Daniel Scholl's avatar
Daniel Scholl committed
35
### Install the Azure CLI
Daniel Scholl's avatar
Daniel Scholl committed
36

Daniel Scholl's avatar
Daniel Scholl committed
37
38
39
40
41
42
43
44
45
46
47
48
49
For information specific to your operating system, see the [Azure CLI install guide](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest). You can also use the single command install if running on a Unix based machine.

```bash
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

# Login to Azure CLI and ensure subscription is set to desired subscription
az login
az account set --subscription <your_subscription>
```


### Configure and Work with an Azure Devops Project

Daniel Scholl's avatar
Daniel Scholl committed
50
> [Role Documentation](https://docs.microsoft.com/en-us/azure/devops/organizations/security/about-security-identity?view=azure-devops): Working on ADO Organizations require certain Roles for the user.  To perform these activities a user must be able to create an ADO project in an organization and have administrator level access to the Project created.
Daniel Scholl's avatar
Daniel Scholl committed
51

Daniel Scholl's avatar
Daniel Scholl committed
52
53
54
Configure an Azure Devops Project in your Organization called `osdu-mvp` and set the cli command to use the organization by default.

```bash
Daniel Scholl's avatar
Daniel Scholl committed
55
export ADO_ORGANIZATION=<organization_name>
Daniel Scholl's avatar
Daniel Scholl committed
56
57
58
59
60
61
export ADO_PROJECT=osdu-mvp

# Ensure the CLI extension is added
az extension add --name azure-devops

# Setup a Project Space in your organization
Daniel Scholl's avatar
Daniel Scholl committed
62
az devops project create --name $ADO_PROJECT --organization https://dev.azure.com/$ADO_ORGANIZATION/
Daniel Scholl's avatar
Daniel Scholl committed
63
64
65
66
67
68
69
70
71
72
73
74
75
76

# Configure the CLI Defaults to work with the organization and project
az devops configure --defaults organization=https://dev.azure.com/$ADO_ORGANIZATION project=$ADO_PROJECT
```

## Clone the repository

It is recommended to work with this repository in a WSL Ubuntu directory structure.

```bash
git clone git@community.opengroup.org:osdu/platform/deployment-and-operations/infra-azure-provisioning.git

cd infra-azure-provisioning
```
Daniel Scholl's avatar
Daniel Scholl committed
77
78


Daniel Scholl's avatar
Daniel Scholl committed
79
## Create a Flux Manifest Repository
Daniel Scholl's avatar
Daniel Scholl committed
80

Daniel Scholl's avatar
Daniel Scholl committed
81
[Create an empty git repository](https://docs.microsoft.com/en-us/azure/devops/repos/git/create-new-repo?view=azure-devops) with a name that clearly signals that the repo is used for the Flux manifests. For example `k8-gitops-manifests`.
Daniel Scholl's avatar
Daniel Scholl committed
82

Daniel Scholl's avatar
Daniel Scholl committed
83
Flux requires that the git repository have at least one commit. Initialize the repo with an empty commit.
Daniel Scholl's avatar
Daniel Scholl committed
84
85

```bash
Daniel Scholl's avatar
Daniel Scholl committed
86
87
88
89
90
91
92
93
94
export ADO_REPO=k8-gitops-manifests

# Initialize a Git Repository
(mkdir k8-gitops-manifests \
  && cd k8-gitops-manifests \
  && git init \
  && git commit --allow-empty -m "Initializing the Flux Manifest Repository")

# Create an ADO Repo
Daniel Scholl's avatar
Daniel Scholl committed
95
az repos create --name $ADO_REPO
Daniel Scholl's avatar
Daniel Scholl committed
96
97
98
99
100
101
export GIT_REPO=git@ssh.dev.azure.com:v3/${ADO_ORGANIZATION}/${ADO_PROJECT}/k8-gitops-manifests

# Push the Git Repository
(cd k8-gitops-manifests \
  && git remote add origin $GIT_REPO \
  && git push -u origin --all)
Daniel Scholl's avatar
Daniel Scholl committed
102
103
```

Daniel Scholl's avatar
Daniel Scholl committed
104
105
106
107
108
109
110
In order for Automated Pipelines to be able to work with this repository the following Permissions must be set in the ADO Project for `All Repositories/Permissions` on the user `osdu-mvp Build Service`.

- Create Branch `Allow`
- Contribute `Allow`
- Contribute to Pull requests `Allow`


Daniel Scholl's avatar
Daniel Scholl committed
111

Daniel Scholl's avatar
Daniel Scholl committed
112
## Provision the Common Resources
Daniel Scholl's avatar
Daniel Scholl committed
113
> [Role Documentation](https://docs.microsoft.com/en-us/azure/role-based-access-control/rbac-and-directory-admin-roles): Provisioning Common Resources requires owner access to the subscription, however AD Service Principals are created that will required an AD Admin to grant approval consent on the principals created.
Daniel Scholl's avatar
Daniel Scholl committed
114

Daniel Scholl's avatar
Daniel Scholl committed
115

Daniel Scholl's avatar
Daniel Scholl committed
116
The script `common_prepare.sh` script is a _helper_ script designed to help setup some of the common things that are necessary for infrastructure.
Daniel Scholl's avatar
Daniel Scholl committed
117

Daniel Scholl's avatar
Daniel Scholl committed
118
119
- Ensure you are logged into the azure cli with the desired subscription set.
- Ensure you have the access to run az ad commands.
Daniel Scholl's avatar
Daniel Scholl committed
120

Daniel Scholl's avatar
Daniel Scholl committed
121
```bash
Daniel Scholl's avatar
Daniel Scholl committed
122
# Execute Script
Daniel Scholl's avatar
Daniel Scholl committed
123
export UNIQUE=demo
Daniel Scholl's avatar
Daniel Scholl committed
124

125
./infra/scripts/common_prepare.sh $(az account show --query id -otsv) $UNIQUE
Daniel Scholl's avatar
Daniel Scholl committed
126
```
Daniel Scholl's avatar
Daniel Scholl committed
127

128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
Integration Tests requires 2 Azure AD User Accounts, a tenant user and a guest user to be setup in order to use for integration testing.  This activity needs to be performed by someone who has access as an AD User Admin.

- ad-guest-email (ie: integration.test@email.com)
- ad-guest-oid (OID of the user)
- ad-user-email (ie: integration.test@{tenant}.onmicrosoft.com
- ad-user-oid (OID of the user)

```bash
USER_EMAIL=""
USER_EMAIL_OID=""
GUEST_EMAIL=""
GUEST_EMAIL_OID=""

az keyvault secret set --vault-name $COMMON_VAULT --name "ad-user-email" --value $USER_EMAIL
az keyvault secret set --vault-name $COMMON_VAULT --name "ad-user-oid" --value $USER_EMAIL_OID
az keyvault secret set --vault-name $COMMON_VAULT --name "ad-guest-email" --value $GUEST_EMAIL
az keyvault secret set --vault-name $COMMON_VAULT --name "ad-guest-oid" --value $GUEST_EMAIL_OID
```

Istio Configuration setups a Dashboard that requires some admin credentials.  Automation Pipelines uses settings out of the common keyvault for applying the values of the Istio Dashboard default credentials.

```bash
ISTIO_USERNAME=""
ISTIO_PASSWORD=""


az keyvault secret set --vault-name $COMMON_VAULT --name "istio-username" --value $(echo $ISTIO_USERNAME |base64)
az keyvault secret set --vault-name $COMMON_VAULT --name "istio-password" --value $(echo $ISTIO_PASSWORD |base64)
```
Daniel Scholl's avatar
Daniel Scholl committed
157

Daniel Scholl's avatar
Daniel Scholl committed
158
__Local Script Output Resources__
Daniel Scholl's avatar
Daniel Scholl committed
159

Daniel Scholl's avatar
Daniel Scholl committed
160
The script creates some local files to be used.
Daniel Scholl's avatar
Daniel Scholl committed
161

Daniel Scholl's avatar
Daniel Scholl committed
162
163
1. .envrc_{UNIQUE} -- This is a copy of the required environment variables for the common components.
2. .envrc -- This file is used directory by direnv and requires `direnv allow` to be run to access variables.
Daniel Scholl's avatar
Daniel Scholl committed
164
165
3. ~/.ssh/osdu_{UNIQUE}/azure-aks-gitops-ssh-key -- SSH key used by flux.
4. ~/.ssh/osdu_{UNIQUE}/azure-aks-gitops-key.pub -- SSH Public Key used by flux.
Jason's avatar
Jason committed
166
167
168
5. ~/.ssh/osdu_{UNIQUE}/azure-aks-node-ssh-key -- SSH Key used by AKS
6. ~/.ssh/osdu_{UNIQUE}/azure-aks-node-ssh-key.pub -- SSH Public Key used by AKS
7. ~/.ssh/osdu_{UNIQUE}/azure-aks-node-ssh-key.passphrase -- SSH Key Passphrase used by AKS
Daniel Scholl's avatar
Daniel Scholl committed
169

Daniel Scholl's avatar
Daniel Scholl committed
170
> Ensure environment variables are loaded `direnv allow`
Daniel Scholl's avatar
Daniel Scholl committed
171

Daniel Scholl's avatar
Daniel Scholl committed
172
__Installed Azure Resources__
Daniel Scholl's avatar
Daniel Scholl committed
173

Daniel Scholl's avatar
Daniel Scholl committed
174
175
176
1. Resource Group
2. Storage Account
3. Key Vault
Daniel Scholl's avatar
Daniel Scholl committed
177
178
179
180
4. A principal to be used by Terraform to create all resources for an OSDU Environment. _(Requires Grant Admin Approval)_
5. A principal required by an OSDU environment deployment that will have root level access to that environment. _(Requires Grant Admin Approval)_
6. An AD application to be leveraged in the future that defines and controls access to the OSDU Environment for AD Identity. _(future)_
7. An AD application to be used for negative integration testing
Daniel Scholl's avatar
Daniel Scholl committed
181

Daniel Scholl's avatar
Daniel Scholl committed
182
> Removal would require deletion of all AD elements `osdu-mvp-{UNIQUE}-*`, unlocking and deleting the resource group.
Daniel Scholl's avatar
Daniel Scholl committed
183
184
185
186
187
188
189
190


__Azure AD Admin Consent__

2 service principals have been created that need to have an AD Admin `grant admin consent` on.

1. osdu-mvp-{UNIQUE}-terraform  _(Azure AD Application Graph - Application.ReadWrite.OwnedBy)_
2. osdu-mvp-{UNIQUE}-principal _(Microsoft Graph - Directory.Read.All)_
Daniel Scholl's avatar
Daniel Scholl committed
191

Jason's avatar
Jason committed
192
193
For more information on Azure identity and authorization, see the official Microsoft documentation [here](https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-permissions-and-consent).

Daniel Scholl's avatar
Daniel Scholl committed
194

Daniel Scholl's avatar
Daniel Scholl committed
195
## Elastic Search Setup
Daniel Scholl's avatar
Daniel Scholl committed
196

Daniel Scholl's avatar
Daniel Scholl committed
197
Infrastructure requires a bring your own Elastic Search Instance of a version of 7.x (ie: 7.11.1) with a valid https endpoint and the access information must now be stored in the Common KeyVault. The recommended method of Elastic Search is to use the [Elastic Cloud Managed Service from the Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/elastic.ec-azure?tab=Overview).
Daniel Scholl's avatar
Daniel Scholl committed
198
199

> Note: Elastic Cloud Managed Service requires a Credit Card to be associated to the subscription for billing purposes.
Daniel Scholl's avatar
Daniel Scholl committed
200

Daniel Scholl's avatar
Daniel Scholl committed
201
```bash
Daniel Scholl's avatar
Daniel Scholl committed
202
203
204
ES_ENDPOINT=""
ES_USERNAME=""
ES_PASSWORD=""
Daniel Scholl's avatar
Daniel Scholl committed
205
206
207
208
az keyvault secret set --vault-name $COMMON_VAULT --name "elastic-endpoint-dp1-${UNIQUE}" --value $ES_ENDPOINT
az keyvault secret set --vault-name $COMMON_VAULT --name "elastic-username-dp1-${UNIQUE}" --value $ES_USERNAME
az keyvault secret set --vault-name $COMMON_VAULT --name "elastic-password-dp1-${UNIQUE}" --value $ES_PASSWORD

Daniel Scholl's avatar
Daniel Scholl committed
209

Daniel Scholl's avatar
Daniel Scholl committed
210
cat >> .envrc << EOF
Daniel Scholl's avatar
Daniel Scholl committed
211

Daniel Scholl's avatar
Daniel Scholl committed
212
213
# https://cloud.elastic.co
# ------------------------------------------------------------------------------------------------------
Daniel Scholl's avatar
Daniel Scholl committed
214
215
216
export TF_VAR_elasticsearch_endpoint="$ES_ENDPOINT"
export TF_VAR_elasticsearch_username="$ES_USERNAME"
export TF_VAR_elasticsearch_password="$ES_PASSWORD"
Daniel Scholl's avatar
Daniel Scholl committed
217

Daniel Scholl's avatar
Daniel Scholl committed
218
EOF
Daniel Scholl's avatar
Daniel Scholl committed
219

Daniel Scholl's avatar
Daniel Scholl committed
220
cp .envrc .envrc_${UNIQUE}
Daniel Scholl's avatar
Daniel Scholl committed
221
```
Daniel Scholl's avatar
Daniel Scholl committed
222

Daniel Scholl's avatar
Daniel Scholl committed
223
## Configure Key Access in Manifest Repository
Daniel Scholl's avatar
Daniel Scholl committed
224

Daniel Scholl's avatar
Daniel Scholl committed
225
The public key of the `azure-aks-gitops-ssh-key` previously created needs to be added as a deploy key in your Azure DevOPS Project, follow these [steps](https://docs.microsoft.com/en-us/azure/devops/repos/git/use-ssh-keys-to-authenticate?view=azure-devops&tabs=current-page#step-2--add-the-public-key-to-azure-devops-servicestfs) to add your public SSH key to your ADO environment.
Daniel Scholl's avatar
Daniel Scholl committed
226

Daniel Scholl's avatar
Daniel Scholl committed
227
228
229
230
231
232
233
```bash
# Retrieve the public key
az keyvault secret show \
  --id https://$COMMON_VAULT.vault.azure.net/secrets/azure-aks-gitops-ssh-key-pub \
  --query value \
  -otsv
```
Daniel Scholl's avatar
Daniel Scholl committed
234

Daniel Scholl's avatar
Daniel Scholl committed
235

236
## Install OSDU
Daniel Scholl's avatar
Daniel Scholl committed
237

238
There are 2 methods that can be chosen to perform installation at this point in time.
Daniel Scholl's avatar
Daniel Scholl committed
239

240
1. Manual Installation -- Typically used when the desire is to manually make modifications to the environment and have full control all updates and deployments.
Daniel Scholl's avatar
Daniel Scholl committed
241

242
2. Pipeline Installation -- Typically used when the need is to only access the Data Platform but allow for automatic upgrades of infrastructure and services.
Daniel Scholl's avatar
Daniel Scholl committed
243

Daniel Scholl's avatar
Daniel Scholl committed
244

MANISH KUMAR's avatar
MANISH KUMAR committed
245
__Manual Installation (**Preferred**)__
Daniel Scholl's avatar
Daniel Scholl committed
246

247
> This typically takes about 2 hours to complete.
Daniel Scholl's avatar
Daniel Scholl committed
248

249
1. Install the Infrastructure following directions [here](./infra/templates/osdu-r3-mvp).
Daniel Scholl's avatar
Daniel Scholl committed
250

Jason's avatar
Jason committed
251
1. Setup DNS to point to the deployed infrastructure following directions [here](./docs/dns-setup.md).
252

Daniel Scholl's avatar
Daniel Scholl committed
253
1. Upload the Configuration Data following directions [here](./docs/configuration-data.md).
254

Daniel Scholl's avatar
Daniel Scholl committed
255
1. Deploy the application helm charts following the directions [here](https://community.opengroup.org/osdu/platform/deployment-and-operations/helm-charts-azure).
256

Daniel Scholl's avatar
Daniel Scholl committed
257
258
1. Upload the Integration Test Data following directions [here](./tools/test_data).

Jason's avatar
Jason committed
259
1. Register your partition with the Data Partition API by following the instructions [here](./tools/rest/README.md) to configure your IDE to make authenticated requests to your OSDU instance and send the API request located [here](./tools/rest/partition.http) (createPartition).
Jason's avatar
Jason committed
260

261
262
1. Load Service Data following directions [here](./docs/service-data.md).

263
264
265
266
267
268
269
270


__Automated Pipeline Installation__

> This typically takes about 3 hours to complete.

1. Setup Code Mirroring following directions [here](./docs/code-mirroring.md).

Jason's avatar
Jason committed
271
1. Setup Infrastructure Automation following directions [here](./docs/infra-automation.md).
Daniel Scholl's avatar
Daniel Scholl committed
272

Jason's avatar
Jason committed
273
1. Setup DNS to point to the deployed infrastructure following directions [here](./docs/dns-setup.md).
Daniel Scholl's avatar
Daniel Scholl committed
274

Daniel Scholl's avatar
Daniel Scholl committed
275
276
1. Upload the Configuration Data following directions [here](./docs/configuration-data.md).

Jason's avatar
Jason committed
277
1. Upload the Integration Test Data following directions [here](./tools/test_data).
278

Jason's avatar
Jason committed
279
1. Setup Service Automation following directions [here](./docs/service-automation.md).
280

281

282
283
284
285
286
287
288
__Data Migration for Entitlements from Milestone 4(v0.7.0) or lower, to Milestone 5(v0.8.0) or higher__

Milestone 5(v0.8.0) introduced a breaking changed for Entitlements, which required migration of data using the scipt
[here](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/tree/master/data-migration).
The script should be run whenever you update OSDU installation from less than Milestone 5(v0.8.0) to equivalent or higher.


Daniel Scholl's avatar
Daniel Scholl committed
289
## Developer Activities
Daniel Scholl's avatar
Daniel Scholl committed
290

Daniel Scholl's avatar
Daniel Scholl committed
291
1. To onboard new services follow the process located [here](./docs/service-onboarding.md).
292
293
294
295
296
297


## Configure Back Up
Back is enabled by default. To set the backup policies, utilize the script
[here](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools).
The script should be run whenever you bring up a Resource Group in your deployment.