README.md 7.53 KB
Newer Older
ethiraj krishnamanaidu's avatar
ethiraj krishnamanaidu committed
1
2
# infra-azure-provisioning

Daniel Scholl's avatar
Daniel Scholl committed
3
This repository contains the infrastructure as code implementation and pipelines necessary for the required infrastructure to host OSDU on Azure.
Dania Kodeih (Microsoft)'s avatar
Dania Kodeih (Microsoft) committed
4

Daniel Scholl's avatar
Daniel Scholl committed
5
The `osdu` - R3 MVP Architecture solution template is intended to provision Managed Kubernetes resources like AKS and other core OSDU cloud managed services like Cosmos, Blob Storage and Keyvault.
Dania Kodeih (Microsoft)'s avatar
Dania Kodeih (Microsoft) committed
6

Daniel Scholl's avatar
Daniel Scholl committed
7
## Cloud Resource Architecture
Dania Kodeih (Microsoft)'s avatar
Dania Kodeih (Microsoft) committed
8

Daniel Scholl's avatar
Daniel Scholl committed
9
![Architecture](./docs/images/architecture.png "Architecture")
Dania Kodeih (Microsoft)'s avatar
Dania Kodeih (Microsoft) committed
10

Dania Kodeih (Microsoft)'s avatar
Dania Kodeih (Microsoft) committed
11

Daniel Scholl's avatar
Daniel Scholl committed
12
## Cost
13

Daniel Scholl's avatar
Daniel Scholl committed
14
Azure environment cost ballpark [estimate](https://tinyurl.com/y4e9s7rf). This is subject to change and is driven from the resource pricing tiers configured when the template is deployed.
15

Daniel Scholl's avatar
Daniel Scholl committed
16

Daniel Scholl's avatar
Daniel Scholl committed
17
## Prerequisites
Daniel Scholl's avatar
Daniel Scholl committed
18

Daniel Scholl's avatar
Daniel Scholl committed
19
20
21
22
1. Azure Subscription
1. Terraform and Go are locally installed.
1. Requires the use of [direnv](https://direnv.net/).
1. Install the required common tools (kubectl, helm, and terraform).  Currently uses [Terraform 0.12.29](https://releases.hashicorp.com/terraform/0.12.29/) and [GO 1.12.14](https://golang.org/dl/).
Daniel Scholl's avatar
Daniel Scholl committed
23
24
25



Daniel Scholl's avatar
Daniel Scholl committed
26
### Install the required tooling
Daniel Scholl's avatar
Daniel Scholl committed
27

Daniel Scholl's avatar
Daniel Scholl committed
28
This document assumes one is running a current version of Ubuntu. Windows users can install the Ubuntu Terminal from the Microsoft Store. The Ubuntu Terminal enables Linux command-line utilities, including bash, ssh, and git that will be useful for the following deployment. _Note: You will need the Windows Subsystem for Linux installed to use the Ubuntu Terminal on Windows_.
Daniel Scholl's avatar
Daniel Scholl committed
29

Daniel Scholl's avatar
Daniel Scholl committed
30
> Note: Terraform and Go are recommended to be installed using a [Terraform Version Manager](https://github.com/tfutils/tfenv) and a [Go Version Manager](https://github.com/stefanmaric/g)
Daniel Scholl's avatar
Daniel Scholl committed
31
32


Daniel Scholl's avatar
Daniel Scholl committed
33
### Install the Azure CLI
Daniel Scholl's avatar
Daniel Scholl committed
34

Daniel Scholl's avatar
Daniel Scholl committed
35
For information specific to your operating system, see the [Azure CLI install guide](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest). You can also use [this script](https://github.com/microsoft/bedrock/blob/master/tools/prereqs/setup_azure_cli.sh) if running on a Unix based machine.
Daniel Scholl's avatar
Daniel Scholl committed
36
37


Daniel Scholl's avatar
Daniel Scholl committed
38
## Create a Flux Manifest Repository
Daniel Scholl's avatar
Daniel Scholl committed
39

Daniel Scholl's avatar
Daniel Scholl committed
40
[Create an empty git repository](https://docs.microsoft.com/en-us/azure/devops/repos/git/create-new-repo?view=azure-devops) with a name that clearly signals that the repo is used for the Flux manifests. For example `k8-gitops-manifests`.
Daniel Scholl's avatar
Daniel Scholl committed
41

Daniel Scholl's avatar
Daniel Scholl committed
42
Flux requires that the git repository have at least one commit. Initialize the repo with an empty commit.
Daniel Scholl's avatar
Daniel Scholl committed
43
44

```bash
Daniel Scholl's avatar
Daniel Scholl committed
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
export ADO_ORGANIZATION=<organization_name>   # ie: osdu-demo
export ADO_PROJECT=<project_name>             # ie: osdu-mvp
export ADO_REPO=k8-gitops-manifests

# Initialize a Git Repository
(mkdir k8-gitops-manifests \
  && cd k8-gitops-manifests \
  && git init \
  && git commit --allow-empty -m "Initializing the Flux Manifest Repository")

# Create an ADO Repo
az repos create --name $ADO_REPO --organization https://dev.azure.com/${ADO_ORGANIZATION} --project $ADO_PROJECT -ojson
export GIT_REPO=git@ssh.dev.azure.com:v3/${ADO_ORGANIZATION}/${ADO_PROJECT}/k8-gitops-manifests

# Push the Git Repository
(cd k8-gitops-manifests \
  && git remote add origin $GIT_REPO \
  && git push -u origin --all)
Daniel Scholl's avatar
Daniel Scholl committed
63
64
```

Daniel Scholl's avatar
Daniel Scholl committed
65

Daniel Scholl's avatar
Daniel Scholl committed
66
## Provision the Common Resources
Daniel Scholl's avatar
Daniel Scholl committed
67

Daniel Scholl's avatar
Daniel Scholl committed
68
The script `common_prepare.sh` script is a _helper_ script designed to help setup some of the common things that are necessary for infrastructure.
Daniel Scholl's avatar
Daniel Scholl committed
69

Daniel Scholl's avatar
Daniel Scholl committed
70
71
- Ensure you are logged into the azure cli with the desired subscription set.
- Ensure you have the access to run az ad commands.
Daniel Scholl's avatar
Daniel Scholl committed
72

Daniel Scholl's avatar
Daniel Scholl committed
73
74
75
76
```bash
# Login to Azure CLI and ensure subscription is set to desired subscription
az login
az account set --subscription <your_subscription>
Daniel Scholl's avatar
Daniel Scholl committed
77

Daniel Scholl's avatar
Daniel Scholl committed
78
# Execute Script
Daniel Scholl's avatar
Daniel Scholl committed
79
UNIQUE=demo
Daniel Scholl's avatar
Daniel Scholl committed
80
./infra/templates/osdu-r3-mvp/common_prepare.sh $(az account show --query id -otsv) $UNIQUE
Daniel Scholl's avatar
Daniel Scholl committed
81
```
Daniel Scholl's avatar
Daniel Scholl committed
82

Daniel Scholl's avatar
Daniel Scholl committed
83
This results in 2 service principals being created that need an AD Admin to `grant admin consent` on.
Daniel Scholl's avatar
Daniel Scholl committed
84

Daniel Scholl's avatar
Daniel Scholl committed
85
86
1. osdu-mvp-{UNIQUE}-terraform
2. osdu-mvp-{UNIQUE}-principal
Daniel Scholl's avatar
Daniel Scholl committed
87
88


Daniel Scholl's avatar
Daniel Scholl committed
89
90
> Removal would require deletion of all AD elements `osdu-mvp-{UNIQUE}-*`, the resource group and purging the KV.

Daniel Scholl's avatar
Daniel Scholl committed
91
__Local Script Output Resources__
Daniel Scholl's avatar
Daniel Scholl committed
92

Daniel Scholl's avatar
Daniel Scholl committed
93
The script creates some local files to be used.
Daniel Scholl's avatar
Daniel Scholl committed
94

Daniel Scholl's avatar
Daniel Scholl committed
95
96
1. .envrc_{UNIQUE} -- This is a copy of the required environment variables for the common components.
2. .envrc -- This file is used directory by direnv and requires `direnv allow` to be run to access variables.
Daniel Scholl's avatar
Daniel Scholl committed
97
98
99
100
101
102
3. ~/.ssh/osdu_{UNIQUE}/azure-aks-gitops-ssh-key -- SSH key used by flux.
4. ~/.ssh/osdu_{UNIQUE}/azure-aks-gitops-key.pub -- SSH Public Key used by flux.
5. ~/.ssh/osdu_{UNIQUE}/azure-aks-gitops-key.passphrase -- SSH Key Passphrase used by flux.
6. ~/.ssh/osdu_{UNIQUE}/azure-aks-node-ssh-key -- SSH Key used by AKS
7. ~/.ssh/osdu_{UNIQUE}/azure-aks-node-ssh-key.pub -- SSH Public Key used by AKS
8. ~/.ssh/osdu_{UNIQUE}/azure-aks-node-ssh-key.passphrase -- SSH Key Passphrase used by AKS
Daniel Scholl's avatar
Daniel Scholl committed
103
104


Daniel Scholl's avatar
Daniel Scholl committed
105
__Installed Common Resources__
Daniel Scholl's avatar
Daniel Scholl committed
106

Daniel Scholl's avatar
Daniel Scholl committed
107
108
109
110
111
112
113
1. Resource Group
2. Storage Account
3. Key Vault
4. A principal to be used for Terraform _(Requires Grant Admin Approval)_
5. A principal to be used for the OSDU environment.
6. An application to be used for the OSDU environment. _(future)_
7. An application to be used for negative integration testing.
Daniel Scholl's avatar
Daniel Scholl committed
114

Daniel Scholl's avatar
Daniel Scholl committed
115
>Note: 2 Users are required to be created manually in AD for integration testing purposes manually and values stored in this Common Key Vault.
Daniel Scholl's avatar
Daniel Scholl committed
116
117
118



Daniel Scholl's avatar
Daniel Scholl committed
119
## Elastic Search Setup
Daniel Scholl's avatar
Daniel Scholl committed
120

Daniel Scholl's avatar
Daniel Scholl committed
121
Infrastructure assumes bring your own Elastic Search Instance at a version of 6.8.x and access information must be stored in the Common KeyVault.
Daniel Scholl's avatar
Daniel Scholl committed
122

Daniel Scholl's avatar
Daniel Scholl committed
123
124
125
126
127
128
129
```bash
ENDPOINT=""
USERNAME=""
PASSWORD=""
az keyvault secret set --vault-name $COMMON_VAULT --name "elastic-endpoint-dp1-demo" --value $ENDPOINT
az keyvault secret set --vault-name $COMMON_VAULT --name "elastic-username-dp1-demo" --value $USERNAME
az keyvault secret set --vault-name $COMMON_VAULT --name "elastic-password-dp1-demo" --value $PASSWORD
Daniel Scholl's avatar
Daniel Scholl committed
130

Daniel Scholl's avatar
Daniel Scholl committed
131
cat >> .envrc << EOF
Daniel Scholl's avatar
Daniel Scholl committed
132

Daniel Scholl's avatar
Daniel Scholl committed
133
134
# https://cloud.elastic.co
# ------------------------------------------------------------------------------------------------------
135
136
137
export TF_VAR_elasticsearch_endpoint="$(az keyvault secret show --id https://$COMMON_VAULT.vault.azure.net/secrets/elastic-endpoint-dp1-demo --query value -otsv)"
export TF_VAR_elasticsearch_username="$(az keyvault secret show --id https://$COMMON_VAULT.vault.azure.net/secrets/elastic-username-dp1-demo --query value -otsv)"
export TF_VAR_elasticsearch_password="$(az keyvault secret show --id https://$COMMON_VAULT.vault.azure.net/secrets/elastic-password-dp1-demo --query value -otsv)"
Daniel Scholl's avatar
Daniel Scholl committed
138

Daniel Scholl's avatar
Daniel Scholl committed
139
EOF
Daniel Scholl's avatar
Daniel Scholl committed
140

Daniel Scholl's avatar
Daniel Scholl committed
141
cp .envrc .envrc_${UNIQUE}
Daniel Scholl's avatar
Daniel Scholl committed
142
```
Daniel Scholl's avatar
Daniel Scholl committed
143
144


Daniel Scholl's avatar
Daniel Scholl committed
145
## Configure Key Access in Manifest Repository
Daniel Scholl's avatar
Daniel Scholl committed
146

Daniel Scholl's avatar
Daniel Scholl committed
147
The public key of the `azure-aks-gitops-ssh-key` previously created needs to be added as a deploy key in your Azure DevOPS Project, follow these [steps](https://docs.microsoft.com/en-us/azure/devops/repos/git/use-ssh-keys-to-authenticate?view=azure-devops&tabs=current-page#step-2--add-the-public-key-to-azure-devops-servicestfs) to add your public SSH key to your ADO environment.
Daniel Scholl's avatar
Daniel Scholl committed
148

Daniel Scholl's avatar
Daniel Scholl committed
149
150
151
152
153
154
155
156
```bash
# Retrieve the public key
az keyvault secret show \
  --vault-name $COMMON_VAULT \
  --id https://$COMMON_VAULT.vault.azure.net/secrets/azure-aks-gitops-ssh-key-pub \
  --query value \
  -otsv
```
Daniel Scholl's avatar
Daniel Scholl committed
157

Daniel Scholl's avatar
Daniel Scholl committed
158

Daniel Scholl's avatar
Daniel Scholl committed
159
## Automated Pipeline Installation
Daniel Scholl's avatar
Daniel Scholl committed
160

Daniel Scholl's avatar
Daniel Scholl committed
161
> This typically takes about 3 hours to complete.
Daniel Scholl's avatar
Daniel Scholl committed
162

Daniel Scholl's avatar
Daniel Scholl committed
163
1. Configure the Pipelines following directions [here](./docs/pipeline-setup.md).
Daniel Scholl's avatar
Daniel Scholl committed
164

Daniel Scholl's avatar
Daniel Scholl committed
165
2. Deploy the application helm charts following the directions [here]().
Daniel Scholl's avatar
Daniel Scholl committed
166

Daniel Scholl's avatar
Daniel Scholl committed
167
3. Load the data
Daniel Scholl's avatar
Daniel Scholl committed
168

Daniel Scholl's avatar
Daniel Scholl committed
169

Daniel Scholl's avatar
Daniel Scholl committed
170
## Manual Installation
Daniel Scholl's avatar
Daniel Scholl committed
171

Daniel Scholl's avatar
Daniel Scholl committed
172
> This typically takes about 2 hours to complete.
Daniel Scholl's avatar
Daniel Scholl committed
173

Daniel Scholl's avatar
Daniel Scholl committed
174
1. Install the Infrastructure following directions [here](./infra/templates/osdu-r3-mvp/README.md).
Daniel Scholl's avatar
Daniel Scholl committed
175

Daniel Scholl's avatar
Daniel Scholl committed
176
2. Deploy the application helm charts following the directions [here](./charts/README.md).
Daniel Scholl's avatar
Daniel Scholl committed
177

Daniel Scholl's avatar
Daniel Scholl committed
178
3. Load the data
Daniel Scholl's avatar
Daniel Scholl committed
179

Daniel Scholl's avatar
Daniel Scholl committed
180

Daniel Scholl's avatar
Daniel Scholl committed
181
## Developer Activities
Daniel Scholl's avatar
Daniel Scholl committed
182

Daniel Scholl's avatar
Daniel Scholl committed
183
1. To onboard new services follow the process located [here](./docs/service-onboarding.md).