Commit cf855454 authored by Aliaksandr Lubouski (EPAM)'s avatar Aliaksandr Lubouski (EPAM) 🥑
Browse files

GONRG-3629: documentation merged and moved to simple-osdu folder

parent 65d9d34f
......@@ -2,109 +2,48 @@
Creates fully functional GCP based OSDU installation.
## Prerequisites
[Simple OSDU installation guide](examples/simple_osdu/README.md)
## OSDU Deployment Framework Status(19.10.2021)
`Disclaimer`: At current state the most stable installation source is master branch. After all services will be migrated to GKE and added to Terraform, there will be release of Deployment Framework and recomended way of installing will be installing release version of the product.
### Services currently added to the Deplyment Framework
* **Backup**
* **Dataset**
* **Entitlements**
* **File**
* **Indexer-Queue**
* **Indexer**
* **Legal**
* **Notification**
* **Partition**
* **Policy**
* **Register**
* **Schema**
* **Search**
* **Seismic-Store**
* **Unit**
* **Wellbore**
* **WKS**
* **Workflow**
* **CRS-Catalog**
* **CRS-Conversion**
### Major Infrastructure Components
* **PostgreSQL**
* **GKE**
* **Anthos Service Mesh**
* **Cloud Buckets**
* **Pub-Sub**
* **Redis**
* **Airflow(Composer)**
* **Cloud Tasks**
* **Cryptographic keys**
* **Service Accounts**
Terraform installs services via `resource "helm_release"` , if needed this charts could be downloaded locally from `repository` parameter of `helm_release`. All terraform service files located at [directory](modules/osdu). Or it's easy to `grep` on repository for files with `helm_release` word.
- **Terraform** (version: v1.0.0) [terraform](https://www.terraform.io/downloads.html)
- **Gcloud** (version: Google Cloud SDK 345.0.0) [googleCloud](https://cloud.google.com/sdk/docs/install)
- **Kubectl** (version: v1.21.0) [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl)
- **kpt** (version: v0.39.3) [github](https://github.com/GoogleContainerTools/kpt/releases/tag/v0.39.3)
- **jq** (version: jq-1.6)
### Manual actions
These actions couldn't be automated at this moment. Should be done once in a new Google Cloud project.
- Enable Cloud Resource Manager API:
- Go to -> `APIs & Services` -> click on `Enable Apis and Services` -> search for `Cloud Resource Manager API` and **enable** this API.
- Get credentials from gcloud command:
- Acquire new user credentials via gcloud command to use for Application default credentials (use this link [cloud.google.com](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login) as an example).
- Create AppEngine app using Google Console GUI:
- Go to -> `App Engine` -> click `Create app` -> choose region corresponding your deployment **Or** use bash script for creating datastore indexes located [here](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-gcp-provisioning/-/blob/master/bootstrap_infra/create-definitions/create-index-definitions.sh)
## Before install
At the respective GCP project, we need to create OAuth 2.0 Client ID.
According to GCP requirement, you need:
- Create `OAuth consent screen`
- Activate `API & Services`
- Create `OAuth 2.0 Client ID (type: Web application)`
### **To create OAuth consent screen:**
Go to **GCP navigation menu** -> **API & Services** -> **OAuth consent screen**
- Choose type `External` and click `CREATE`
- Fill in these fields (you can fild example below):
- `App name`
- `User support email`
- `Authorized domains`
- `Developer contact information`
- Press now and later `SAVE AND CONTINUE`
![alt text](img/gcp-consent-screen.png "Content Screen")
### **To create OAuth 2.0 Client ID of type Web application:**
- Go to **GCP navigation menu** -> **API & Services** -> **Credentials**
- Click `+ CREATE CREDENTIALS`
- Use `OAuth client ID`
- Application type (choose `Web application`)
- Fill in `Name` field like `project_name-audiences`
- Add Authorized redirect URIs:
- `http://localhost:8080/auth/callback`
- `https://developers.google.com/oauthplayground/`
- Click `CREATE`
![alt text](img/web-credentials.png "Web application credentials")
## How to run **osdu-module** with **Terraform**
Make sure that your gcloud client is configured with appropriate GCP **project** and **region**\
Use this links as examples:
- Set default [project](https://cloud.google.com/artifact-registry/docs/gcloud-defaults)
- Set default [region](https://cloud.google.com/compute/docs/gcloud-compute#set_default_zone_and_region_in_your_local_client)
### To start installation, use the following commands
```\
- git clone git@community.opengroup.org:osdu/platform/deployment-and-operations/infra-gcp-provisioning.git
- cd infra-gcp-provisioning/examples/simple_osdu/
```
### Then you need to fill in **mandatory** variables at **variables.tf** file
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
**project_id** | your GCP project ID | string | none | yes
**project_region** | your GCP project region | string | none | yes
**project_zone** | your GCP project zone | string | none | yes
**data_partition_id** | Data Partition ID and also Tenant Name | string | **dataid** | yes
**domain** | DNS name for OSDU installation | string | none | yes
**elastic_pass** | password for ElasticSearch instance | string | none | yes
**elastic_host** | domain name of ElasticSearch instance | string | none | yes
**audiences** | this variable is ClientID from previous step of creating OAuth 2.0 Client ID | string | none | yes
**admin_user_email** | Admin person user email in project or at google identity service | string | **user@example.com** | yes
> **data_partition_id** variable could contain only lowercase letters and numbers, and couldn't be longer than 25 characters.
>
> **elastic_host** variable might look like: instance-name.es.us-central1.cloud.es.io
>
> **audiences** variable could look like: 689762842995-pv21xxxxxxx803kk6gqf52qb5amos3a9.apps.googleusercontent.com
Then perform following commands:
```\
terraform init
terraform plan
terraform apply
```
You can provide variables by `-var` option when running the terraform plan and terraform apply commands.\
Example: `terraform apply -var="domain=example.com"`
## Post install steps
After successfull installation, look at the **outputs** of terraform execution and
get **ingress_ip** variable value for usage this ip-address in DNS domain name
......@@ -7,18 +7,21 @@ This example illustrates how to create an osdu set of services within a single G
### Packages
Packages are only needed for installation from local computer. Please use Google Cloud Shell as it is preconfigured and contains all the required dependencies and software.
- kpt [GitHub](https://github.com/GoogleContainerTools/kpt/releases)
- jq [GitHub](https://github.com/stedolan/jq/releases)
- gcloud [Google Cloud](https://cloud.google.com/sdk/docs/install)
- kubectl [Kubernetes.io](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)
- **Terraform** (version: v1.0.0 or higher) [terraform](https://www.terraform.io/downloads.html)
- **Gcloud** (version: Google Cloud SDK 345.0.0 or higher) [googleCloud](https://cloud.google.com/sdk/docs/install)
- **Kubectl** (version: v1.21.0 or higher) [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl)
- **kpt** (version: v0.39.3) [github](https://github.com/GoogleContainerTools/kpt/releases/tag/v0.39.3)
- **jq** (version: jq-1.6 or higher)
### Elasticsearch
OSDU requires Elasticsearch engine.
Recomended version for ElasticSearch is **7.10 or higher**. Currently on 18.10.2021, version **8** is not supported.
It can be configured as managed service at [Elactic Cloud](https://cloud.elastic.co/) or on-premises.
From the OSDU module perspective it is vital to get elasticsearch FQDN and password to provision infrastructure.
User name is hardcoded as "elastic" (default Elasticsearch user name).
### Manual actions
These actions couldn't be automated at this moment. Should be done once in a new Google Cloud project.
- Open Google Cloud Project console. Go to "APIs & Services"
......@@ -33,19 +36,61 @@ These actions couldn't be automated at this moment. Should be done once in a new
![Enable API](images/Eojoh3ai.png)
![Enable API](images/Oboo2gai.png)
- Activate Google Cloud Shell in Google Cloud Console
### **OAuth 2.0 Client ID Creation**
At the respective GCP project, we need to create OAuth 2.0 Client ID.
According to GCP requirement, you need:
- Create `OAuth consent screen`
- Activate `API & Services`
- Create `OAuth 2.0 Client ID (type: Web application)`
#### **To create OAuth consent screen:**
Go to **GCP navigation menu** -> **API & Services** -> **OAuth consent screen**
- Choose type `External` and click `CREATE`
- Fill in these fields (you can fild example below):
- `App name`
- `User support email`
- `Authorized domains`
- `Developer contact information`
- Press now and later `SAVE AND CONTINUE`
![alt text](img/gcp-consent-screen.png "Content Screen")
#### **To create OAuth 2.0 Client ID of type Web application:**
- Go to **GCP navigation menu** -> **API & Services** -> **Credentials**
- Click `+ CREATE CREDENTIALS`
- Use `OAuth client ID`
- Application type (choose `Web application`)
- Fill in `Name` field like `project_name-audiences`
- Add Authorized redirect URIs:
- `http://localhost:8080/auth/callback`
- `https://developers.google.com/oauthplayground/`
- Click `CREATE`
![alt text](img/web-credentials.png "Web application credentials")
### OSDU Installation Prerequisites:
To start installation, use the following commands and steps.
- **Activate Google Cloud Shell in Google Cloud Console**
![Activate Cloud Shell](images/cof0Tai3.png)
- Clone this repository into Cloud Shell:
- **Clone this repository into Cloud Shell:**
git clone git@community.opengroup.org:osdu/platform/deployment-and-operations/infra-gcp-provisioning.git
$ git clone git@community.opengroup.org:osdu/platform/deployment-and-operations/infra-gcp-provisioning.git
- Change directory with "cd" command to infra-gcp-provisioning/example/simple-osdu
- **Change directory with "cd" command to infra-gcp-provisioning/example/simple-osdu**
cd infra-gcp-provisioning/example/simple-osdu
$ cd infra-gcp-provisioning/example/simple-osdu
- Set variables in file variables.tf using any code editor.
- **Set variables in file variables.tf using any code editor.**
- **project_id** - id of your google project;
- **project_region** - your project region, e.g. "us-central1";
- **project_zone** - your project zone, e.g. "us-central1-c";
......@@ -53,17 +98,30 @@ These actions couldn't be automated at this moment. Should be done once in a new
- **domain** - domain, you are going to use for OSDU installation;
- **elastic_pass** - password to your elasticsearch instance;
- **elastic_host** - FQDN of your elastic instance;
- **audiences** - id of your GCP audiences, e.g. "763519302943-n9310sjjr8aaf4to6r5lfjsr4is6kjb4.apps.googleusercontent.com". More information about this variable can be found [here](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-gcp-provisioning/-/blob/master/README.md#to-create-oauth-consent-screen);
- **audiences** - id of your GCP audiences, e.g. "763519302943-n9310xxxxxto6r5lfjsr4is6kjb4.apps.googleusercontent.com". To obtain this variable please refer to previous step -> **To create OAuth 2.0 Client ID of type Web application**;
- **admin_user_email** - admin person user email in project or at google identity service.
Variables can also be provided in command line using `-var` option when running the terraform plan and terraform apply commands:
terraform apply -var="domain=example.com"
## Inputs
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
### TBD
**project_id** | your GCP project ID | string | none | yes
**project_region** | your GCP project region | string | none | yes
**project_zone** | your GCP project zone | string | none | yes
**data_partition_id** | Data Partition ID and also Tenant Name | string | **dataid** | yes
**domain** | DNS name for OSDU installation | string | none | yes
**elastic_pass** | password for ElasticSearch instance | string | none | yes
**elastic_host** | domain name of ElasticSearch instance | string | none | yes
**audiences** | this variable is ClientID from previous step of creating OAuth 2.0 Client ID | string | none | yes
**admin_user_email** | Admin person user email in project or at google identity service | string | **user@example.com** | yes
> **data_partition_id** variable could contain only lowercase letters and numbers, and couldn't be longer than 25 characters.
>
> **elastic_host** variable might look like: instance-name.es.us-central1.cloud.es.io
>
> **audiences** variable could look like: 689762842995-pv21xxxxxxx803kk6gqf52qb5amos3a9.apps.googleusercontent.com
## Outputs
......@@ -95,9 +153,10 @@ Then use Actions field three vertical dots, and choose `Manage keys`, proceed wi
Go to `Cloud Storage` -> find bucket with a name ending with `-airflow-sa-bucket` click on this bucket
use `UPLOAD FILES` button, and upload service account json file to the bucket.
After successfull installation, look at the **outputs** of terraform execution and
get **ingress_ip** variable value for usage this ip-address in DNS domain name
Point your domain name to IP address obtained in the output if terraform apply step.
#### Deploy Osdu_ingest DAG
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment