Update OSDU QSG authored by Dzmitry Malkevich (EPAM)'s avatar Dzmitry Malkevich (EPAM)
......@@ -54,19 +54,19 @@ _Steps:_
- Navigate to "Cloud Functions" > "postman-function-python" > "TRIGGER"
- Click on Trigger URL (image 1)
As a result, the _gcp-.postman_environment.json_ file should be downloaded.
As a result, the _gcp-.postman_environment.json_ file should be downloaded.
![image001](uploads/55b6ba43691be1d8e651167decebbc1d/image001.png)
![Image 1 – Cloud Function. Trigger URL](uploads/55b6ba43691be1d8e651167decebbc1d/image001.png)
<div>Image 1 – Cloud Function. Trigger URL</div>2. \\\\\\\*\\\\\\\*Download Postman\\\\\\\*\\\\\\\*
2. **Download Postman**
Download and install Postman (for example the [link](https://www.postman.com/downloads) may be used).
Download and install Postman (for example the [link](https://www.postman.com/downloads) may be used).
3. **Import the environment variables file into Postman**
![image003](uploads/921b240be55455b4fca5010337962ce1/image003.png)
![Image2 – Postman. Import of environment variables](uploads/921b240be55455b4fca5010337962ce1/image003.png)
<div>Image2 – Postman. Import of environment variables,</div>Import previously downloaded \\\\\\\_gcp-.postman_environment.json\\\\\\\_ to Postman Environments.
Import previously downloaded _gcp-.postman_environment.json_ to Postman Environments.
### Grant permissions for users
......@@ -80,25 +80,78 @@ The instruction for granting permissions is available by [the link](https://comm
### Set a value for **_refresh_token_**
You have several options to obtain `refresh_token`. But in all cases you need to have CLIENT_ID and CLIENT_SECRET and set them in Postman environment.
- Ask “Owner” or “Editor” of the GoogleCloud project where OSDU is deployed to create a client that is planned to use (API & Services -> Credentials -> OAuth 2.0 Client IDs)
![Image 4. Adding a client](uploads/3548c5785436951575a471cc04d05fd5/image007.png)
![Image 6.GCP: Client ID and Client Secret](uploads/8b0592b7bea223f84e20f9b0efdadf43/image011.png)
- Copy-paste CLIENT_ID and CLIENT_SECRET into the environmental variables in Postman. Make sure you are updating both `INITIAL VALUE` and `CURRENT VALUE`
![Image 5. Updating CLIEND_ID and CLIENT_SECRET in the Postman environment variables](uploads/f984c539a3ade847147ad3872a171e7c/image009.png)
#### Obtaining `refresh_token` via Postman UI
Refer to [Postman authentication guide](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M16/GCP-M16/Postman_Authentication_Guide.md).
Open [Quick start](https://community.opengroup.org/osdu/documentation/-/wikis/uploads/3c9820ad7dfe9ed873755c53500afe4e/OSDU_Quick_start.postman_collection.json) or other Postman collection and go to the Authorization tab:
![auth tab](./uploads/refi-qsg-img/kk-postman-auth-1.png)
Verify that following variables are set in Postman enviroment:
```text
callback_url=https://developers.google.com/oauthplayground
auth_url=https://accounts.google.com/o/oauth2/auth
Token_Fetch_URL=https://oauth2.googleapis.com/token
Scope=email openid profile
```
Set `Type=OAuth 2.0` and scoll down to `Configure New Token` and press `Edit token configuration`.
Set `Token Name`, `Grant Type=Authorization Code`, `Callback URL={{callback_url}}`, `Auth URL={{auth_url}}?access_type=offline&prompt=consent`
`Access Token URL={{Token_Fetch_URL}}`, `Client ID={{CLIENT_ID}}`, `Client Secret{{CLIENT_SECRET}}`, `Scope={{Scope}}`.
![auth](./uploads/qsg-img/postman-auth.png)
Press `Get New Access Token`.
In opened window enter your Google account email
![login](./uploads/qsg-img/postman-auth-login.png)
and password on next step
![password](./uploads/qsg-img/postman-auth-password.png)
Allow access on next step:
![allow access](./uploads/qsg-img/postman-auth-access.png)
In `Manage access tokens` window scroll down to `refresh_token` section, select this value and copy it. Than paste in your Postman environent:
![tokens](./uploads/qsg-img/postman-tokens.png)
Save the environment.
Now you could use `Refresh Token` requests from Postman collection to get new access tokens.
#### Obtaining `refresh_token` via Google OAuth 2.0 Playground
- Open [OAuth 2.0 Playground](https://developers.google.com/oauthplayground)
- In Step 1 “Select and authorize APIs” expand the “Google OAuth2 API v2” section and select all 3 values (image 3):
- https://www.googleapis.com/auth/userinfo.email
- https://www.googleapis.com/auth/userinfo.profile
- openid
- Click on “OAuth 2.0 Configuration” icon and select “Use your own OAuth credentials” (image 3) ![image005](uploads/4c4e0f7a8f3d8af88fe949ec00d8be80/image005.png)
- Ask “Owner” or “Viewer” of the GCP project where OSDU is deployed to create a client (Image 4) that is planned to use (API & Services -> Credentials -> OAuth 2.0 Client IDs) ![image007](uploads/3548c5785436951575a471cc04d05fd5/image007.png)
<div>Image 4. Adding a client</div>- Copy-paste CLIENT_ID and CLIENT_SECRET into the environmental variables in Postman (Image 5). Make sure you are updating both \\\\\\\*\\\\\\\*INITIAL VALUE\\\\\\\*\\\\\\\* and \\\\\\\*\\\\\\\*CURRENT VALUE\\\\\\\*\\\\\\\* ![image009](uploads/f984c539a3ade847147ad3872a171e7c/image009.png)
<div>Image 5. Updating CLIEND_ID and CLIENT_SECRET in the Postman environment variables</div>- Click on the needed client and copy-paste CLIENT_ID and CLIENT_SECRET into the previous OAuth 2.0 Playground screen and click “Authorize APIs”. ![image011](uploads/8b0592b7bea223f84e20f9b0efdadf43/image011.png)
<div>Image 6.GCP: Client ID and Client Secret</div>- ![image013](uploads/a2d6f51dc38ee820b5701c1f27138528/image013.png)
- Click on “OAuth 2.0 Configuration” icon and select “Use your own OAuth credentials” (image 3)
![image005](uploads/4c4e0f7a8f3d8af88fe949ec00d8be80/image005.png)
<div>Image 7.OAuth 2.0 Playground. Updating Client ID and Client Secret</div>- Select a user account that can be authorized on the next screen - Click “Exchange authorization code for tokens” (Image 8) ![image015](uploads/ea2fdfb596224d002bdfe460626595e7/image015.png)
- Paste CLIENT_ID and CLIENT_SECRET from above step to OAuth 2.0 Playground screen and click “Authorize APIs”.
![Image 7.OAuth 2.0 Playground. Updating Client ID and Client Secret](uploads/a2d6f51dc38ee820b5701c1f27138528/image013.png)
<div>Image 8. OAuth 2.0 Playground. Exchange authorization code for tokens</div>- Copy refresh token into the Postman environment variables files (both \\\\\\\*\\\\\\\*INITIAL VALUE\\\\\\\*\\\\\\\* and \\\\\\\*\\\\\\\*CURRENT VALUE\\\\\\\*\\\\\\\*) (Image 9) ![image017](uploads/9d8496589ec9cd47da4833ac61e72745/image017.png)
- Select a user account that can be authorized on the next screen - Click “Exchange authorization code for tokens” (Image 8)
![Image 8. OAuth 2.0 Playground. Exchange authorization code for tokens](uploads/ea2fdfb596224d002bdfe460626595e7/image015.png)
<div>Image 9.OAuth 2.0 Playground. Refresh token</div>
- Copy `refresh token` into the Postman environment variables files (both `INITIAL VALUE` and `CURRENT VALUE`) (Image 9)
![Image 9. OAuth 2.0 Playground. Refresh token](uploads/9d8496589ec9cd47da4833ac61e72745/image017.png)
Remember that **the tokens should be refreshed in Postman every 30 minutes.**
......@@ -308,7 +361,7 @@ Metadata is described by the [JSON schemas](https://gitlab.opengroup.org/osdu/su
![image019](uploads/68cf061e56f57e4106feadba00db5175/image019.png)
<div>Image 10. Data (Files) ingestion \\\\\\\\\\\\\\\\ download</div>A user can choose 1 of 4 ways to upload data into OSDU.
<div>Image 10. Data (Files) ingestion \\ download</div>A user can choose 1 of 4 ways to upload data into OSDU.
Let’s see how we can ingest Raster Well Log file in .tif format into OSDU. Please note that you can ingest a file of any format.
......@@ -372,13 +425,13 @@ Parameter “FileSource” is stored as {{file_source}} variable.
Request:
```
```http
PUT {{upload_signed_url}}
```
Response:
```
```http
Status: 200 OK
```
......@@ -412,7 +465,9 @@ There are several ways of storing metadata into OSDU:
![image029](uploads/c526a4dd508bef50585cdbc5169c87f8/image029.png)
<div>Image 15. Metadata Ingestion</div>#### Example: Metadata Ingestion using File Service API
<div>Image 15. Metadata Ingestion</div>
#### Example: Metadata Ingestion using File Service API
**_Pre-requisites:_** Example: Data Ingestion using File Service API should be executed
......@@ -470,7 +525,7 @@ POST {{file_api_url}}/v2/files/metadata
_The response:_
```
```http
Status 201 Created
{
"id": "odesprod:dataset--File.Generic:ce6fe9fd-ab46-4358-ae27-8631e6cf8ae4"
......@@ -585,7 +640,7 @@ A mapping document between the .tif file and OSDU schema has to be created:
![image035](uploads/fcada9d0bf68c088921f1d6bb0d9ef26/image035.png)
<div>Image 16. A mapping document</div>\\\\\\\*\\\\\\\*Step 2. Create a manifest file\\\\\\\*\\\\\\\*
<div>Image 16. A mapping document</div>**Step 2. Create a manifest file**
The structure of the manifest is described by [the OSDU Manifest schema](https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/blob/master/Generated/manifest/Manifest.1.0.0.json)
......@@ -1037,14 +1092,13 @@ To review the log, please navigate to your Airflow console and open osdu_ingest
_Request:_
```
```http
GET {{storage_api_url}}/api/storage/v2/records/odesprod:work-product-component--WellLog:f8781f4e41b04b96b385d70a0f8a14bf
```
_Response:_
```json
{
"data": {
"Description": "Raster Well Log",
......@@ -1103,7 +1157,7 @@ See below a couple of simple search queries.
#### Example: Search for a metadata record by id using Storage Service API
```
```http
GET https://{{STORAGE_HOST}}/records/{{record-id}}
```
......@@ -1126,7 +1180,7 @@ First, you need to search for the needed metadata record of the dataset (see abo
#### Example: Get file DownloadUrl using File Service API
```
```http
GET {{FILE_HOST}}/files/{{record_id}}/downloadURL
```
......@@ -1139,4 +1193,4 @@ Postman collection file: [OSDU_Quick_start.postman_collection.json](uploads/3c98
- [OSDU API Quick start demo](https://gitlab.opengroup.org/osdu/pmc/docs/-/blob/master/Google%20Cloud/Quick_Start_Guide_demo.mp4)
- [create OSDU on-prem using Helm chart](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-gcp-provisioning/-/blob/master/examples/simple_osdu_onprem/README.md)
- [create OSDU set of services within a single Google Cloud project](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-gcp-provisioning/-/blob/master/examples/simple_osdu/README.md)
- [create OSDU set of service within Azure](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/README.md)
\ No newline at end of file
- [create OSDU set of service within Azure](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/README.md)