Update OSDU QSG authored by Dzmitry Malkevich (EPAM)'s avatar Dzmitry Malkevich (EPAM)
...@@ -56,17 +56,17 @@ _Steps:_ ...@@ -56,17 +56,17 @@ _Steps:_
As a result, the _gcp-.postman_environment.json_ file should be downloaded. As a result, the _gcp-.postman_environment.json_ file should be downloaded.
![image001](uploads/55b6ba43691be1d8e651167decebbc1d/image001.png) ![Image 1 – Cloud Function. Trigger URL](uploads/55b6ba43691be1d8e651167decebbc1d/image001.png)
<div>Image 1 – Cloud Function. Trigger URL</div>2. \\\\\\\*\\\\\\\*Download Postman\\\\\\\*\\\\\\\* 2. **Download Postman**
Download and install Postman (for example the [link](https://www.postman.com/downloads) may be used). Download and install Postman (for example the [link](https://www.postman.com/downloads) may be used).
3. **Import the environment variables file into Postman** 3. **Import the environment variables file into Postman**
![image003](uploads/921b240be55455b4fca5010337962ce1/image003.png) ![Image2 – Postman. Import of environment variables](uploads/921b240be55455b4fca5010337962ce1/image003.png)
<div>Image2 – Postman. Import of environment variables,</div>Import previously downloaded \\\\\\\_gcp-.postman_environment.json\\\\\\\_ to Postman Environments. Import previously downloaded _gcp-.postman_environment.json_ to Postman Environments.
### Grant permissions for users ### Grant permissions for users
...@@ -80,25 +80,78 @@ The instruction for granting permissions is available by [the link](https://comm ...@@ -80,25 +80,78 @@ The instruction for granting permissions is available by [the link](https://comm
### Set a value for **_refresh_token_** ### Set a value for **_refresh_token_**
You have several options to obtain `refresh_token`. But in all cases you need to have CLIENT_ID and CLIENT_SECRET and set them in Postman environment.
- Ask “Owner” or “Editor” of the GoogleCloud project where OSDU is deployed to create a client that is planned to use (API & Services -> Credentials -> OAuth 2.0 Client IDs)
![Image 4. Adding a client](uploads/3548c5785436951575a471cc04d05fd5/image007.png)
![Image 6.GCP: Client ID and Client Secret](uploads/8b0592b7bea223f84e20f9b0efdadf43/image011.png)
- Copy-paste CLIENT_ID and CLIENT_SECRET into the environmental variables in Postman. Make sure you are updating both `INITIAL VALUE` and `CURRENT VALUE`
![Image 5. Updating CLIEND_ID and CLIENT_SECRET in the Postman environment variables](uploads/f984c539a3ade847147ad3872a171e7c/image009.png)
#### Obtaining `refresh_token` via Postman UI
Refer to [Postman authentication guide](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M16/GCP-M16/Postman_Authentication_Guide.md).
Open [Quick start](https://community.opengroup.org/osdu/documentation/-/wikis/uploads/3c9820ad7dfe9ed873755c53500afe4e/OSDU_Quick_start.postman_collection.json) or other Postman collection and go to the Authorization tab:
![auth tab](./uploads/refi-qsg-img/kk-postman-auth-1.png)
Verify that following variables are set in Postman enviroment:
```text
callback_url=https://developers.google.com/oauthplayground
auth_url=https://accounts.google.com/o/oauth2/auth
Token_Fetch_URL=https://oauth2.googleapis.com/token
Scope=email openid profile
```
Set `Type=OAuth 2.0` and scoll down to `Configure New Token` and press `Edit token configuration`.
Set `Token Name`, `Grant Type=Authorization Code`, `Callback URL={{callback_url}}`, `Auth URL={{auth_url}}?access_type=offline&prompt=consent`
`Access Token URL={{Token_Fetch_URL}}`, `Client ID={{CLIENT_ID}}`, `Client Secret{{CLIENT_SECRET}}`, `Scope={{Scope}}`.
![auth](./uploads/qsg-img/postman-auth.png)
Press `Get New Access Token`.
In opened window enter your Google account email
![login](./uploads/qsg-img/postman-auth-login.png)
and password on next step
![password](./uploads/qsg-img/postman-auth-password.png)
Allow access on next step:
![allow access](./uploads/qsg-img/postman-auth-access.png)
In `Manage access tokens` window scroll down to `refresh_token` section, select this value and copy it. Than paste in your Postman environent:
![tokens](./uploads/qsg-img/postman-tokens.png)
Save the environment.
Now you could use `Refresh Token` requests from Postman collection to get new access tokens.
#### Obtaining `refresh_token` via Google OAuth 2.0 Playground
- Open [OAuth 2.0 Playground](https://developers.google.com/oauthplayground) - Open [OAuth 2.0 Playground](https://developers.google.com/oauthplayground)
- In Step 1 “Select and authorize APIs” expand the “Google OAuth2 API v2” section and select all 3 values (image 3): - In Step 1 “Select and authorize APIs” expand the “Google OAuth2 API v2” section and select all 3 values (image 3):
- https://www.googleapis.com/auth/userinfo.email - https://www.googleapis.com/auth/userinfo.email
- https://www.googleapis.com/auth/userinfo.profile - https://www.googleapis.com/auth/userinfo.profile
- openid - openid
- Click on “OAuth 2.0 Configuration” icon and select “Use your own OAuth credentials” (image 3) ![image005](uploads/4c4e0f7a8f3d8af88fe949ec00d8be80/image005.png) - Click on “OAuth 2.0 Configuration” icon and select “Use your own OAuth credentials” (image 3)
- Ask “Owner” or “Viewer” of the GCP project where OSDU is deployed to create a client (Image 4) that is planned to use (API & Services -> Credentials -> OAuth 2.0 Client IDs) ![image007](uploads/3548c5785436951575a471cc04d05fd5/image007.png) ![image005](uploads/4c4e0f7a8f3d8af88fe949ec00d8be80/image005.png)
<div>Image 4. Adding a client</div>- Copy-paste CLIENT_ID and CLIENT_SECRET into the environmental variables in Postman (Image 5). Make sure you are updating both \\\\\\\*\\\\\\\*INITIAL VALUE\\\\\\\*\\\\\\\* and \\\\\\\*\\\\\\\*CURRENT VALUE\\\\\\\*\\\\\\\* ![image009](uploads/f984c539a3ade847147ad3872a171e7c/image009.png)
<div>Image 5. Updating CLIEND_ID and CLIENT_SECRET in the Postman environment variables</div>- Click on the needed client and copy-paste CLIENT_ID and CLIENT_SECRET into the previous OAuth 2.0 Playground screen and click “Authorize APIs”. ![image011](uploads/8b0592b7bea223f84e20f9b0efdadf43/image011.png)
<div>Image 6.GCP: Client ID and Client Secret</div>- ![image013](uploads/a2d6f51dc38ee820b5701c1f27138528/image013.png)
<div>Image 7.OAuth 2.0 Playground. Updating Client ID and Client Secret</div>- Select a user account that can be authorized on the next screen - Click “Exchange authorization code for tokens” (Image 8) ![image015](uploads/ea2fdfb596224d002bdfe460626595e7/image015.png) - Paste CLIENT_ID and CLIENT_SECRET from above step to OAuth 2.0 Playground screen and click “Authorize APIs”.
![Image 7.OAuth 2.0 Playground. Updating Client ID and Client Secret](uploads/a2d6f51dc38ee820b5701c1f27138528/image013.png)
<div>Image 8. OAuth 2.0 Playground. Exchange authorization code for tokens</div>- Copy refresh token into the Postman environment variables files (both \\\\\\\*\\\\\\\*INITIAL VALUE\\\\\\\*\\\\\\\* and \\\\\\\*\\\\\\\*CURRENT VALUE\\\\\\\*\\\\\\\*) (Image 9) ![image017](uploads/9d8496589ec9cd47da4833ac61e72745/image017.png) - Select a user account that can be authorized on the next screen - Click “Exchange authorization code for tokens” (Image 8)
![Image 8. OAuth 2.0 Playground. Exchange authorization code for tokens](uploads/ea2fdfb596224d002bdfe460626595e7/image015.png)
<div>Image 9.OAuth 2.0 Playground. Refresh token</div> - Copy `refresh token` into the Postman environment variables files (both `INITIAL VALUE` and `CURRENT VALUE`) (Image 9)
![Image 9. OAuth 2.0 Playground. Refresh token](uploads/9d8496589ec9cd47da4833ac61e72745/image017.png)
Remember that **the tokens should be refreshed in Postman every 30 minutes.** Remember that **the tokens should be refreshed in Postman every 30 minutes.**
...@@ -308,7 +361,7 @@ Metadata is described by the [JSON schemas](https://gitlab.opengroup.org/osdu/su ...@@ -308,7 +361,7 @@ Metadata is described by the [JSON schemas](https://gitlab.opengroup.org/osdu/su
![image019](uploads/68cf061e56f57e4106feadba00db5175/image019.png) ![image019](uploads/68cf061e56f57e4106feadba00db5175/image019.png)
<div>Image 10. Data (Files) ingestion \\\\\\\\\\\\\\\\ download</div>A user can choose 1 of 4 ways to upload data into OSDU. <div>Image 10. Data (Files) ingestion \\ download</div>A user can choose 1 of 4 ways to upload data into OSDU.
Let’s see how we can ingest Raster Well Log file in .tif format into OSDU. Please note that you can ingest a file of any format. Let’s see how we can ingest Raster Well Log file in .tif format into OSDU. Please note that you can ingest a file of any format.
...@@ -372,13 +425,13 @@ Parameter “FileSource” is stored as {{file_source}} variable. ...@@ -372,13 +425,13 @@ Parameter “FileSource” is stored as {{file_source}} variable.
Request: Request:
``` ```http
PUT {{upload_signed_url}} PUT {{upload_signed_url}}
``` ```
Response: Response:
``` ```http
Status: 200 OK Status: 200 OK
``` ```
...@@ -412,7 +465,9 @@ There are several ways of storing metadata into OSDU: ...@@ -412,7 +465,9 @@ There are several ways of storing metadata into OSDU:
![image029](uploads/c526a4dd508bef50585cdbc5169c87f8/image029.png) ![image029](uploads/c526a4dd508bef50585cdbc5169c87f8/image029.png)
<div>Image 15. Metadata Ingestion</div>#### Example: Metadata Ingestion using File Service API <div>Image 15. Metadata Ingestion</div>
#### Example: Metadata Ingestion using File Service API
**_Pre-requisites:_** Example: Data Ingestion using File Service API should be executed **_Pre-requisites:_** Example: Data Ingestion using File Service API should be executed
...@@ -470,7 +525,7 @@ POST {{file_api_url}}/v2/files/metadata ...@@ -470,7 +525,7 @@ POST {{file_api_url}}/v2/files/metadata
_The response:_ _The response:_
``` ```http
Status 201 Created Status 201 Created
{ {
"id": "odesprod:dataset--File.Generic:ce6fe9fd-ab46-4358-ae27-8631e6cf8ae4" "id": "odesprod:dataset--File.Generic:ce6fe9fd-ab46-4358-ae27-8631e6cf8ae4"
...@@ -585,7 +640,7 @@ A mapping document between the .tif file and OSDU schema has to be created: ...@@ -585,7 +640,7 @@ A mapping document between the .tif file and OSDU schema has to be created:
![image035](uploads/fcada9d0bf68c088921f1d6bb0d9ef26/image035.png) ![image035](uploads/fcada9d0bf68c088921f1d6bb0d9ef26/image035.png)
<div>Image 16. A mapping document</div>\\\\\\\*\\\\\\\*Step 2. Create a manifest file\\\\\\\*\\\\\\\* <div>Image 16. A mapping document</div>**Step 2. Create a manifest file**
The structure of the manifest is described by [the OSDU Manifest schema](https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/blob/master/Generated/manifest/Manifest.1.0.0.json) The structure of the manifest is described by [the OSDU Manifest schema](https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/blob/master/Generated/manifest/Manifest.1.0.0.json)
...@@ -1037,14 +1092,13 @@ To review the log, please navigate to your Airflow console and open osdu_ingest ...@@ -1037,14 +1092,13 @@ To review the log, please navigate to your Airflow console and open osdu_ingest
_Request:_ _Request:_
``` ```http
GET {{storage_api_url}}/api/storage/v2/records/odesprod:work-product-component--WellLog:f8781f4e41b04b96b385d70a0f8a14bf GET {{storage_api_url}}/api/storage/v2/records/odesprod:work-product-component--WellLog:f8781f4e41b04b96b385d70a0f8a14bf
``` ```
_Response:_ _Response:_
```json ```json
{ {
"data": { "data": {
"Description": "Raster Well Log", "Description": "Raster Well Log",
...@@ -1103,7 +1157,7 @@ See below a couple of simple search queries. ...@@ -1103,7 +1157,7 @@ See below a couple of simple search queries.
#### Example: Search for a metadata record by id using Storage Service API #### Example: Search for a metadata record by id using Storage Service API
``` ```http
GET https://{{STORAGE_HOST}}/records/{{record-id}} GET https://{{STORAGE_HOST}}/records/{{record-id}}
``` ```
...@@ -1126,7 +1180,7 @@ First, you need to search for the needed metadata record of the dataset (see abo ...@@ -1126,7 +1180,7 @@ First, you need to search for the needed metadata record of the dataset (see abo
#### Example: Get file DownloadUrl using File Service API #### Example: Get file DownloadUrl using File Service API
``` ```http
GET {{FILE_HOST}}/files/{{record_id}}/downloadURL GET {{FILE_HOST}}/files/{{record_id}}/downloadURL
``` ```
... ...
......