README.md 4.32 KB
Newer Older
Oleksandr Kosse (EPAM)'s avatar
Oleksandr Kosse (EPAM) committed
1
2
3
4
# Simple OSDU

This example illustrates how to create an osdu set of services within a single GCP project.

5

Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
6
7
8
## Prerequisites

### Packages
Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
9
Packages are only needed for installation from local computer. Please use Google Cloud Shell as it is preconfigured and contains all the required dependencies and software.
Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
10
11
12
- kpt [GitHub](https://github.com/GoogleContainerTools/kpt/releases)
- jq [GitHub](https://github.com/stedolan/jq/releases)
- gcloud [Google Cloud](https://cloud.google.com/sdk/docs/install)
13
- kubectl [Kubernetes.io](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)
Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
14

15
16
### Elasticsearch
OSDU requires Elasticsearch engine.
Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
17
It can be configured as managed service at [Elactic Cloud](https://cloud.elastic.co/) or on-premises.
18
19
20
From the OSDU module perspective it is vital to get elasticsearch FQDN and password to provision infrastructure.
User name is hardcoded as "elastic" (default Elasticsearch user name). 

Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
21
22
### Manual actions
These actions couldn't be automated at this moment. Should be done once in a new Google Cloud project.
Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
23
24
- Open Google Cloud Project console. Go to "APIs & Services"

Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
25
26
![APIs & Services](images/TW9uIFNl.png)

Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
27
28
- Click on "Enable Apis and Services" - search for "Cloud Resource Manager API" and enable this API.

29
30
31
32
33
34
35
![Enable Apis and Services](images/Kaeb8tos.png)

- Search for "Cloud Resource Manager API" and enable this API.

![Enable API](images/Eojoh3ai.png)
![Enable API](images/Oboo2gai.png)

36
37
38
39
40
41
42
- Activate Google Cloud Shell in Google Cloud Console

![Activate Cloud Shell](images/cof0Tai3.png)

- Clone this repository into Cloud Shell:

        git clone git@community.opengroup.org:osdu/platform/deployment-and-operations/infra-gcp-provisioning.git
Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
43
44
45

- Change directory with "cd" command to infra-gcp-provisioning/example/simple-osdu

46
47
48
49
50
51
52
53
54
55
        cd infra-gcp-provisioning/example/simple-osdu

- Set variables in file variables.tf using any code editor.
    - **project_id** - id of your google project;
    - **project_region** - your project region, e.g. "us-central1";
    - **project_zone** - your project zone, e.g. "us-central1-c";
    - **data_partition_id** your data partition, e.g. "osdu";
    - **domain** - domain, you are going to use for OSDU installation;
    - **elastic_pass** - password to your elasticsearch instance;
    - **elastic_host** - FQDN of your elastic instance;
Oleksii Tsyganov (EPAM)'s avatar
Typo    
Oleksii Tsyganov (EPAM) committed
56
    - **audiences** - id of your GCP audiences, e.g. "763519302943-n9310sjjr8aaf4to6r5lfjsr4is6kjb4.apps.googleusercontent.com". More information about this variable can be found [here](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-gcp-provisioning/-/tree/GONRG-3390-gcp-osdu-step-by-step-howto-install/#to-create-oauth-consent-screen);
57
    - **admin_user_email** - admin person user email in project or at google identity service.
Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
58
59
60

Variables can also be provided in command line using `-var` option when running the terraform plan and terraform apply commands:
    terraform apply -var="domain=example.com"
Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
61

Oleksandr Kosse (EPAM)'s avatar
Oleksandr Kosse (EPAM) committed
62
63
64
65
66
67
68
69
70
71
72
73
## Inputs

| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
### TBD

## Outputs

| Name | Description |
|------|-------------|
### TBD

Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
74
## Provisioning
Oleksandr Kosse (EPAM)'s avatar
Oleksandr Kosse (EPAM) committed
75

Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
76
To provision this example, run the following from within this directory in Cloud Shell:
Oleksandr Kosse (EPAM)'s avatar
Oleksandr Kosse (EPAM) committed
77
- `terraform init` to get the plugins
Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
78
79

![terraform init step](images/epe8iMej.png)
Oleksandr Kosse (EPAM)'s avatar
Oleksandr Kosse (EPAM) committed
80
- `terraform plan` to see the infrastructure plan
Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
81
82
83
- `terraform apply` to apply the infrastructure build. Confirm applying with `yes`.

![terraform apply step](images/Yae7ohf9.png)
Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
84
85

As a result you will get fully provisioned infrastructure and services.
Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
86
Note the IP address from the output of the command.
Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
87
88

![Provisioning result](images/uaTh7quo.png)
89

90
91
92
93
94
### Post install steps
Go to `IAM & Admin` -> `Service Accounts` -> find service account with a name `osdu-sa-airflow-composer`
Then use Actions field three vertical dots, and choose `Manage keys`, proceed with `ADD KEY` button
`Create New Key`, key type pick `JSON`, click `CREATE`. Service account json file will be dowloaded to your PC. 

Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
95
Go to `Cloud Storage` -> find bucket with a name ending with `-airflow-sa-bucket` click on this bucket
Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
96
97
use `UPLOAD FILES` button, and upload service account json file to the bucket.

Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
98
99
Point your domain name to IP address obtained in the output if terraform apply step.

Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
100
## Destroying
Oleksii Tsyganov (EPAM)'s avatar
Oleksii Tsyganov (EPAM) committed
101
To destroy built infrastructure run `terraform destroy` and confirm it with `yes`.