README.md 14.5 KB
Newer Older
1
# OSDU GCP Infrastructure Provisioning
ethiraj krishnamanaidu's avatar
ethiraj krishnamanaidu committed
2

3
4
5
6
7
8
9
10
11
12
13
14
15
16
## About
This document is the OSDU Google Cloud Platform (GCP) installation manual. It describes the required steps to configure a new GCP project for OSDU.

The main goal is to run all scripts in [this repository](https://github.com/google/framework-for-osdu.git). The order in which you run the scripts matters, since they lay out the infrastructure on a GCP project. For instance, you shall run the `1_enable_apis.sh` script before others, because GCP requires us to enable APIs before using them.

All proposed scripts can be run from a local computer as well as from a release environment (e.g. a Jenkins CI or Gitlab) where you can run these scripts sequentially. This is primarily because the scripts take a long time to run (~20-30 mins, because they do a LOT of stuff), have a couple of manual steps in between because we have to interact with GSuite admin console, and a release environment provides granular control over the executed steps in case of failures/retries.

From security perspective, it is recommended to have a jump host with appropriate permissions for deploying all required services.

## Installation

### Prerequisites
Before running the automated scripts, perform the following steps manually:

17
18
19
20
21
22
1. Define your primary domain name. You need to have enough permissions for adding new records to verify domain ownership and create a subdomain for OSDU
2. Configure [Cloud Identity](https://cloud.google.com/identity/docs/overview) on top of it. Another option is using [GSuite](https://gsuite.google.com/) from your organization, but it is not possible to use free/trial GSuite due to the limit of accounts numbers.
3. Create a [GCP project](https://cloud.google.com/resource-manager/docs/creating-managing-projects)
4. Enable [billing](https://cloud.google.com/billing/docs)
5. Install [Google Cloud SDK](https://cloud.google.com/sdk/) (version 280 or above), [Golang](https://golang.org/), `jq`, `python3-dev`, and `python3-venv`
6. ECE/ESS Cluster Deployment for Elasticsearch. Currently this is a manual procedure, but it is automated internally with the indexer service
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201

    For data indexing and searching, we will use Elasticsearch. We can use either the Elastic Cloud Enterprise (ECE - preffered) solution or the Hosted Elastic Search (ESS) solution. This is an independent resource creation which needs separate billing.

    Another option is using dedicated Elasticsearch instances or Kubernetes application from [Google Marketplace](https://console.cloud.google.com/marketplace), or installed by [Elasticsearch Helm Chart](https://github.com/elastic/helm-charts/tree/master/elasticsearch)

    From the platform perspective it is mandatory to get from Elasticsearch service https endpoint with a trusted certificate.

### Preparation

From the project owner terminal or via [console](https://console.cloud.google.com):

1. Activate the [App Engine](https://cloud.google.com/appengine):
```shell
gcloud app create --region=us-central
```

2. Before running the scripts, create a service account `domain-creator` with a private key json:
```shell
gcloud iam service-accounts create domain-creator \
  --description="domain-creator" \
  --display-name="domain-creator"
gcloud iam service-accounts keys create ~/domain-creator.json \
  --iam-account domain-creator@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com

```
Later you need to add base64 value of that json as the env variable `DOMAIN_CREATOR_SERVICE_ACCOUNT_KEY`

3. Configure [Serverless VPC Access](https://cloud.google.com/vpc/docs/configure-serverless-vpc-access):
```shell
gcloud services enable vpcaccess.googleapis.com && \
  gcloud compute networks vpc-access connectors create vpc-connector \
    --network default \
    --region us-central1  \
    --range 10.8.0.0/28
```

4. Create a builder (builder@<project-name>.iam.gserviceaccount.com) service account with the following roles:
```shell
alias add-role="gcloud projects add-iam-policy-binding ${GOOGLE_CLOUD_PROJECT} \
  --member serviceAccount:builder@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com --role"
add-role roles/bigquery.dataOwner
add-role roles/cloudkms.admin
add-role roles/cloudkms.cryptoKeyEncrypterDecrypter
add-role roles/deploymentmanager.editor
add-role roles/iam.serviceAccountAdmin
add-role roles/resourcemanager.projectIamAdmin
add-role roles/runtimeconfig.admin
add-role roles/serviceusage.serviceUsageAdmin
add-role roles/storage.admin
add-role roles/redis.admin
add-role roles/appengine.appAdmin
add-role roles/datastore.owner
add-role roles/logging.configWriter
add-role roles/cloudtasks.queueAdmin
add-role roles/pubsub.admin
```

5. Create a jump host with the created service accounts to control VM access:
    - `n1-standard-1` as the machine type
    - Debian as a recommended operation system
```shell
gcloud beta compute --project=${GOOGLE_CLOUD_PROJECT} \
  instances create jumphost \
  --zone=us-central1-a \
  --machine-type=n1-standard-1 \
  --subnet=default \
  --service-account=builder@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com \
  --scopes=https://www.googleapis.com/auth/cloud-platform \
  --image=$(gcloud compute images list --uri | grep debian-10) \
  --boot-disk-size=10GB \
  --boot-disk-type=pd-standard \
  --boot-disk-device-name=jumphost
```

### GCP Configuration Steps for OSDU

All steps below should be performed in the jump host unless it recommends different:

1. Install dependencies:
```shell
sudo apt update && \
  sudo apt install python3-venv python3-dev git jq -y
```
2. Download and install [Go tools](https://golang.org/doc/install). Installation scripts are compatible with Golang version 1.10.x.

2. Clone this repository and change the folder:
```shell
git clone https://github.com/google/framework-for-osdu.git
cd framework-for-osdu/osdu-r2/os-infra-provisioning
```
3. Preconfigure all variables except the release-time variables. [These](#environment-variables) are detailed instructions on all ENV variables you need for the release definition. All provided below variables are examples:
```shell
export GCLOUD_PROJECT=<GCP_Project_ID>
export GOOGLE_PROJECT_REGION=us-central1
export CLOUDSDK_COMPUTE_ZONE=us-central1-c
export GCLOUD_REGION=us-central
export ENVIRONMENT_NAME=dev
export DATA_PARTITION_ID=osdu
export TENANT_PROJECT=${GCLOUD_PROJECT}
export COMPLIANCE_RULE_SET=shared
export CRM_ACCOUNT_IDS="osdu"
export NEW_TENANT=true
export DOMAIN_CREATOR_SERVICE_ACCOUNT_KEY=$(cat ../domain.json | base64 -w 0)
export GOOGLE_DOMAIN="example.com"
export GOOGLE_CLOUD_IDENTITY_VERIFIER_EMAIL="opendes_verifier@example.com"
export GOOGLE_CLOUD_IDENTITY_ADMIN_EMAIL="opendes_admin@example.com"
export GSUITE_GROUPS_BOOTSTRAP=true
export ELASTIC_ADMIN_USER=elastic
export ELASTIC_ADMIN_PASSWORD=<ELASTIC_ADMIN_PASSWORD>
export ELASTIC_PORT=<ELASTIC_PORT>
export ELASTIC_HOST=10.128.0.10
export PREFIX=$(pwd)
```
4. Enable [Cloud APIs](https://cloud.google.com/apis) so they can be used:
```shell
cd 1_enable_apis && ./1_enable_apis.sh && cd -
```
5. Create all required service accounts:
```shell
cd 2_service_accounts && ./2_service_accounts.sh && cd -
```
6. Create [Memorystore](https://cloud.google.com/memorystore) deployments for caching:
```shell
./create-redis-instance.sh
```
7. Create [Cloud Key Management Service](https://cloud.google.com/kms) entries:
```shell
cd 3_create_cryptographic_key && ./3_create_cryptographic_key.sh && cd -
```
8. Generate service account keys, put them in buckets, and change IAM permissions:
```shell
cd 4_generate_service_account_key && ./4_generate_service_account_key.sh && cd -
```
9. Create [BigQuery](https://cloud.google.com/bigquery) sink and dataset:
```shell
cd 5_bq_sink_dataset && ./5_bq_sink_dataset.sh && cd -
```
10. Check the tenant name length:
```shell
./validate-tenant-name.sh
```
11. Create [Datastore](https://cloud.google.com/datastore) entities:
```shell
cd create-datastore-entities && ./create-datastore-entity.sh && cd -
```
12. Create buckets for data:
```shell
./create-buckets.sh
```
13. Get list of tables from Datastore:
```shell
cd read-datastore-table && ./read-datastore-table.sh && cd -
```
14. Create a queue in the [Cloud Tasks](https://cloud.google.com/tasks):
```shell
cd create-indexer-queue && ./create-queue.sh && cd -
```
15. Create [Cloud Pub/Sub](https://cloud.google.com/pubsub) topics and subscribers:
```shell
cd create-pub-sub && \
./create-topic.sh && \
./create-subscriptions.sh && \
cd -
```
16. Create indexes in the Cloud Datastore:
```shell
cd create-definitions && ./create-index-definitions.sh && cd -
```
17. Create a master service account:
```shell
./add-master-service-account.sh
```
18. Configure Elasticsearch:
```shell
cd elastic-search-settings && ./create-search-service-user.sh && cd -
```
19. MANUAL STEP:
    1. Create an internal application at Cloud Console APIs&Services OAuth consent [screen](https://console.cloud.google.com/apis/credentials/consent)
    2. Enable DwD on service accounts `datafier`, `domain-creator`, `entitlements-gsuite`, `entitlements-gsuite-init`, `entitlements-gsuite-0`, `entitlements-gsuite-1`, `entitlements-gsuite-2`. This will generate a Client ID for each one
202
    3. Get the `datafier` service account Client ID from GCP console and give it `https://www.googleapis.com/auth/devstorage.full_control` and `https://www.googleapis.com/auth/cloud-platform` OAuth scopes in [G Suite](https://admin.google.com/ac/owl?authuser=0).
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
    4. Get the `domain-creator` service account Client ID from GCP console and give it
    `https://www.googleapis.com/auth/admin.directory.domain`, `https://www.googleapis.com/auth/siteverification` OAuth scope in G Suite
    5. Give the Client ID of service accounts `entitlements-gsuite`, `entitlements-gsuite-init`, `entitlements-gsuite-0`, `entitlements-gsuite-1`, `entitlements-gsuite-2` the following scope in G Suite:
        - `https://www.googleapis.com/auth/admin.directory.domain.readonly`
        - `https://www.googleapis.com/auth/admin.directory.group`
        - `https://www.googleapis.com/auth/admin.directory.group.member`
        - `https://www.googleapis.com/auth/admin.directory.group.member.readonly`
        - `https://www.googleapis.com/auth/admin.directory.group.readonly`
        - `https://www.googleapis.com/auth/admin.directory.user`
    6. Create the users `opendes_verifier` and `opendes_admin` in G Suite and grant them admin permissions
    7. (if not deployed) Deploy a 'default' service to GAE (this is a pre-req by appengine, only then you can deploy other services)
20. Create a subdomain and activate it:
```shell
cd create-subdomain && ./create-subdomain.sh && cd -
```
  Sometimes this step fails and requires a restart or [manual subdomain activation](https://support.google.com/cloudidentity/answer/7331243?hl=en).

21. Generate entitlements default groups and users:
```shell
cd datalake-groups-init && ./create-default-groups.sh && cd -
```
22. Provide permissions for datafier serice account:
```shell
./datafier-permissions.sh
```
228
229
230
231
232
233
234
23. Create gcp.dastore rows for ingestion-strategy (refer docs inside directory):
```shell
cd ingestion-strategy
python ingest-strategy.py --namespace=opendes-namespace --dagname=Osdu_ingest --datatype=osdu  --workflowtype=osdu
python ingest-strategy.py --namespace=opendes-namespace --dagname=Well_log_ingest --datatype=well_log  --workflowtype=ingest
python ingest-strategy.py --namespace=opendes-namespace --dagname=Default_ingest --workflowtype=ingest
```
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286

---
## Environment Variables

### Preconfigured Variables
These variables have a set value and will rarely change

|ENV_VAR_NAME                    |ENV_VAR_VALUE        |ENV_VAR_SCOPE|
-------------                    |----------------     |-------------|
|1. CLOUDSDK_CORE_DISABLE_PROMPTS|   1                 |   Release   |
|2. ELASTIC_ADMIN_USER           |   elastic           |   Release   |
|3. ELASTIC_PPORT                |   9243              |   Release   |
|4. GSUITE_GROUPS_BOOTSTRAP      |   true              |   Release   |
|5. NEW_TENANT                   |   true              |   Release   |
|6. COMPLIANCE_RULE_SET          |   shared            |   Release   |

### G Suite Domain Specific Environment Variables
G Suite domain setup is configured separately from GCP, so the same domain can be used by multiple OSDU ecosystems. Therefore, it is better to store these variables in a variable group for sharing purpose.

1. GOOGLE_CLOUD_IDENTITY_ADMIN_EMAIL: noreply@example.com
2. GOOGLE_DOMAIN: example.com
3. GOOGLE_CLOUD_IDENTITY_VERIFIER_EMAIL: noreply@example.com

### Other Shareable Variables
Other variables that can be shared between micro-services pipelines can also be stored in variable groups. This is optional and chosen only to avoid duplicating the variables usage across different release pipelines.

1. GCLOUD_PROJECT: Google Project ID
2. CLOUDSDK_COMPUTE_ZONE: GCP compute zone (for example, `us-central1-a`)
3. ENVIRONMENT: This is to specify if this GCP is for a prod or a non-prod OSDU environment. Its value can either be `dev`, `evt`, or `p4d` for non prod environments and `demo`, `prod-us`, or `prod-eu` for prod environments
4. GCLOUD_REGION: App engine region (for example, `us-central`)
5. GOOGLE_AUDIENCES: Follow the below steps to generate this
    * On the GCP, go to `APIs & Services` -> `Credentials`
    * Click on `Create credentials` -> `OAuth client ID`
    * Select `Web application` as application type and enter a name (for example, `osdu-service-audience`)
    * Under `Authorized redirect URIs`, add the following value: <br> https://developers.google.com/oauthplayground
    * Click `Create`
    * The value of the Client ID that gets generated is `GOOGLE_AUDIENCES`

### Release-Time Variables
Insert these variables when creating a new release for the release definition.

1. DATA_PARTITION_ID: In multitenancy terms, this is a separate partition which will have a 1-1 mapping to a tenant GCP. For all practical purposes, for our use case, this can be the same as GCP Project ID
2. ELASTIC_ADMIN_PASSWORD: Default password returned by Elastic right after creating a cluster
3. ELASTIC_HOST: Host endpoint of elastic cluster. Remove `https://` from the beginning of this URL and port `:9243` from the ending
4. TENANT_PROJECT: This is same as the GCP Project ID

All these variables should be ticked `Settable at release time` since their values are set release time.

### Other Misc Variables
Any other variables that are specific to this pipeline should be configured in the Pipeline variables section with the proper scope.

1. DOMAIN_CREATOR_SERVICE_ACCOUNT_KEY: This is the base64 encoded value of the service account json key file for service account `domain-creator` in gcp. This value must be kept secret.
287
2. ENVIRONMENT_NAME: Can have either `dev`, `testing`, `p4d`, `demo`, `prod-eu`, or `prod-us` value