-[Purpose of the document](#purpose-of-the-document)
-[GC vs baremetal deployment](#gc-vs-baremetal-deployment)
-[Configuring a Postman environment](#configuring-a-postman-environment)
-[Working with OSDU](#working-with-osdu)
-[Schema creation](#schema-creation)
-[Example: Schema creation](#schema-creation)
-[Example: Record creation with kind](#example-record-creation-with-kind)
-[Legal Tag creation](#legal-tag-creation)
-[Example: Legal Tag creation](#example-legal-tag-creation)
-[Access Control Lists (ACL) configuration](#access-control-lists-acl-configuration)
-[Example: A new data group creation](#example-a-new-data-group-creation)
-[Ingestion](#ingestion)
-[Data Ingestion](#data-ingestion)
-[Example: Data ingestion using Cloud Tools](#example-data-ingestion-using-cloud-tools)
-[Example: Data Ingestion using File Service API](#example-data-ingestion-using-file-service-api)
-[Metadata ingestion](#metadata-ingestion)
-[Example: Metadata Ingestion using File Service API](#example-metadata-ingestion-using-file-service-api)
-[Example: Metadata Ingestion using Dataset Service API](#example-metadata-ingestion-using-dataset-service-api)
-[Example: Metadata Ingestion using Manifest-based ingestion](#example-metadata-ingestion-using-manifest-based-ingestion)
-[Search and Metadata retrieval](#search-and-metadata-retrieval)
-[Example: Search for a metadata record by id using Storage Service API](#example-search-for-a-metadata-record-by-id-using-storage-service-api)
-[Example: Search for a metadata record by id using Search Service API](#example-search-for-a-metadata-record-by-id-using-search-service-api)
-[Download the original file](#download-the-original-file)
-[Example: Get file DownloadUrl using File Service API](#example-get-file-downloadurl-using-file-service-api)
-[Next Steps](#next-steps)
We are moving to this documentation to this new location
### Purpose of the document
The document goals are:
1. to introduce you to the basic OSDU functionality,
2. to help configure a Postman environment to do initial testing,
3. to explain the logic of included Postman requests.
This document is not intended to introduce you to all OSDU services. For full OSDU documentation please use [the link](https://osduforum.org/getting-started/osdu-documentation/)
### GC vs baremetal deployment
This guide is for GC deployment. If you are on baremetal please refer to [this guide](https://community.opengroup.org/osdu/documentation/-/wikis/OSDU-baremetal-QSG) to configure Postman and manage users and their permissions. Then continue with [Working with OSDU](#working-with-osdu) section.
### Configuring a Postman environment
OSDU uses the Postman tool to do majority of API testing. Here are the pre-requisites and the steps you need to perform to configure Postman environment.
_Pre-requisites:_
1. Deployed OSDU on GCP deployed with [examples](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-gcp-provisioning/-/tree/release/0.23/examples/simple_osdu)(for release **M20/v0.23**) or newer ones
2. A valid Google account that is used during OSDU deployment (used as the "admin_user_email" Terraform variable)
_Steps:_
1.**Download Postman environment file for the OSDU installation:**
Since release M19/v0.22 GC installation comes with `config` service that allows download of Postman environment file. Simply use link `https://<your_domain>/api/config/v1/postman-environment` and save the page.
2.**Download Postman**
Download and install Postman (for example the [link](https://www.postman.com/downloads) may be used).
3.**Import the environment variables file into Postman**
Import previously downloaded _gcp-.postman_environment.json_ to Postman Environments.
### Grant permissions for users
An OSDU admin (an email address that was specified as "admin_user_email" during Terraform deployment) grants Entitlements permissions for users who send requests using Postman. These users should be added into the following Entitlements groups:
- users
- users.datalake.viewers
The instruction for granting permissions is available by [the link](https://gitlab.opengroup.org/osdu/pmc/home/-/wikis/Releases/R3.0/GCP/GCP-Operation/User-Mng/User-Management).
### Set a value for **_refresh_token_**
You have several options to obtain `refresh_token`. But in all cases you need to have CLIENT_ID and CLIENT_SECRET and set them in Postman environment.
- For non-production purposes you could use following values:
- For production environments or due to other restrictions ask “Owner” or “Editor” of the GoogleCloud project where OSDU is deployed to create a client that is planned to use (OAuth configuration is out of scope of this document)
- Copy-paste CLIENT_ID and CLIENT_SECRET into the environmental variables in Postman. Make sure you are updating both `INITIAL VALUE` and `CURRENT VALUE`
#### Option 1 (Simple): Obtaining `refresh_token` via Postman UI
Refer to [Postman authentication guide](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M16/GCP-M16/Postman_Authentication_Guide.md).
Open [Quick start](https://community.opengroup.org/osdu/documentation/-/wikis/uploads/3c9820ad7dfe9ed873755c53500afe4e/OSDU_Quick_start.postman_collection.json) or other Postman collection and go to the Authorization tab:
Now you just need to ensure that `Authorization` for all requests and folders in collection are set to `Inherit auth from parent` and Postman will refresh token for you when needed
Remember that **the tokens should be refreshed in Postman every 30 minutes.**
The next chapters will briefly explain several simple and widely used scenarios. For full details on how different OSDU services work, please refer to documentation of a specific OSDU service.
### Working with OSDU
### Schema creation
Schema - A data model definition. In other words, the schema defines whether a given field in the record is a string, or an integer, or a float, or a geopoint, etc. Schema Service allows data models to be defined in rich JSON objects.
Schemas and records are tied together by the **kind** attribute - **kind** of data being ingested. **Kind** value is schema id and must follow the naming convention: _{Schema-Authority}:{dataset-name}:{record-type}:{version}_. On top of that, a given kind can have zero or exactly one schema associated with.
Legal Tags are needed for OSDU metadata access control (more information on the metadata ingestion is later in this document).
To create a Legal Tag, Legal service should be used.
#### Example: Legal Tag creation
```json
POSThttps://{{LEGAL_HOST}}/legaltags
{
"name":"{{data-partition-id}}-demo-legaltag",
"description":"Legal Tag added for Well",
"properties":{
"contractId":"123456",
"countryOfOrigin":[
"US",
"CA"
],
"dataType":"Third Party Data",
"exportClassification":"EAR99",
"originator":"Schlumberger",
"personalData":"No Personal Data",
"securityClassification":"Private",
"expirationDate":"2025-12-25"
}
}
```
### Access Control Lists (ACL) configuration
ACLs are needed to manage user access to the specific metadata record. They define a data group that has “owners” or “viewers” permissions over this metadata record.
If the deployment process and the Entitlements service bootstrapping are performed properly, all the users can be included into 2 groups:
- data-default-viewers
- data-default-owners
However, it's possible to create a new data group that would allow only specific users to view or modify records.
It is easier to start explaining OSDU functionality with explanation on how to ingest data into OSDU.
There are 2 types of data ingestion in OSDU:
1.**Data (files) ingestion** — any file types.
2.**Metadata ingestion** - data that describes the ingested file to make it searchable
Metadata is described by the [JSON schemas](https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/tree/master/Generated) created by the OSDU Data Definition team.
Users can access OSDU Storage Service using cloud tools and just upload needed files there. Here is an example of how to do this on Google Cloud (GC). This way is not supported in baremetal deployment.
**Step 1. Load file(s)**
The Storage Service has a specified bucket in the OSDU environment, and a user can directly upload a file into this bucket. **<span dir="">_TBD – how to get the name of the bucket (to delete?):_</span>**
Parameter “SignedURL” is stored as {{upload_signed_url}} variable.
Parameter “FileSource” is stored as {{file_source}} variable.
**Step 2. Upload your file**
Request:
```http
PUT {{upload_signed_url}}
```
Response:
```http
Status: 200 OK
```
As a result of this step File is moved into OSDU Staging zone (Compute area).
### Metadata ingestion
To make data (ingested files) discoverable (searchable by OSDU Search service) metadata should be stored to OSDU.
All metadata that can be stored into OSDU is described by [OSDU schemas](https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/tree/master/Generated).
<div>Image 14. OSDU Data Platform core concepts and Group Types</div>In simple words:
-**Master data** – slow-changing o&g entities, this data doesn’t have association with the specific file. Full list of OSDU supported Master data can be found [here](https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/tree/master/Generated/master-data).
-**Work Product Component (WPC)** – metadata associated with the specific file or file collection. It is changing in time data, thus WPC may have different versions, each version describing the snapshot of the data at a certain point of time. WPC must have a reference to one or more Dataset records that describe the file or file collection. Full list of supported WPCs can be found [here](https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/tree/master/Generated/work-product-component).
-**Reference data** – think about it as a drop-down values to describe Master or WPC attributes values (e.g. units of measures, list of countries, etc.). By governance type reference data can be:
- fixed (values are provided by OSDU
- open (values are provided by OSDU, but an operating company can add their own values)
- local (schema is provided by OSDU, but values should be provided by an operating company).
See the full list of reference data schemas with their governance type [here](https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/tree/master/E-R/reference-data).
-**Dataset** – schemas that describe properties of a physical file (this file does not describe the logical entity!). They can describe files of the specific type (e.g. .tiff, .segy) or any type (File.Generic kind). Each Dataset record must have a link to the physical file location that can be used to download the file from the OSDU Storage service. Schemas that describe file properties can be found [here](https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/tree/master/E-R/dataset).
-**Work Product (WP)** – think of it as a project or collection of related files and datasets. Work Product is an abstract entity that allows to organize different WPCs and datasets. You can have any number of WPCs and Datasets in your WP.
There are several ways of storing metadata into OSDU:
Please note that the id parameter is not specified in the request, and it was assigned automatically by the Storage Service. You may choose to specify the “id” parameter in case you want to assign a specific id.
As a result of this step the file is moved from the Staging area to the Persistent area and metadata record for the file is created.
### Example: Metadata Ingestion using Dataset Service API
Files and File Collections can be ingested into OSDU using Dataset Service.
### Example: Metadata Ingestion using Manifest-based ingestion
Metadata can be ingested using OSDU Manifest-based ingestion mechanism. Depending on the data type included in your file, the corresponding OSDU Schema should be used.
All OSDU schemas can be found [here](https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/tree/master/Generated).
In our example we are ingesting Well Log file, so we will use [the Well Log schema](https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/blob/master/Generated/work-product-component/WellLog.1.0.0.json).
**Step 1. Create a mapping between the file that contains metadata information and an OSDU schema**
As was stated above, we are using OSDU Well Log schema. Metadata to populate this schema should be taken from the .tif file itself. Please note that sometimes metadata about the file may be sourced from another file or a database.
A mapping document between the .tif file and OSDU schema has to be created:
<div>Image 16. A mapping document</div>In this case we have an excel spreadsheet with the mapping. Also, some of the schema attributes can be hardcoded:
<div>Image 16. A mapping document</div>**Step 2. Create a manifest file**
The structure of the manifest is described by [the OSDU Manifest schema](https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/blob/master/Generated/manifest/Manifest.1.0.0.json)
The Well Log schema is embedded into the Manifest schema.
You can populate manifest manually or using a custom script.
Here are some places in manifest that user should pay attention to:
1. Master data, Reference data, Work Product (WP), Work Product Component (WPC), Dataset ids
Depending on your use case, your organization may want to specify ids for WP, WPC and Dataset or use system-generated ids.
- Assigned ids: Use “id” parameter in the manifest following the pattern defined in the Manifest schema:
For Master and Reference data ingestion, you just can skip “id” field in the manifest, in this case id value will be assigned automatically by the Storage Service.
For WP ingestion (e.g. Well Log) you need to use surrogate-keys. Surrogate-keys are defined as an “id” value for WP, WPC and Dataset according to the pattern defined in the Manifest schema. For example:
```json
"Data":{
"WorkProduct":{
"id":"surrogate-key:wp-1",
```
2. WPC and Dataset references
Please make sure that all WPCs are referenced in the corresponding WP block (all WPC ids are listed). Ids should end with ":" symbol:
Please make sure that all Datasets are referenced in the corresponding WPC block (all Dataset ids are listed). Ids should end with ":" symbol. For example:
If Master data, Reference data or WPC is referenced in the manifest, id should end with the : symbol (to reference the latest version of the record) or with : to reference a specific version of the record. For example:
4. Make sure that dataset record is created / enriched correctly
Depending on the way you chose to ingest file(s) in Part 1 of this document you may have a metadata record for a Dataset (if you used File Service or Dataset Service APIs) or you may not have a metadata record (you used Cloud Tools to load data).
- File metadata exists
If a file metadata was ingested during file loading, current metadata should enrich existing record. To do so, you need to specify the record id created in Part 1 as a dataset id in the manifest.
Make sure that you are also specifying correct FileSource parameter that references file location in the OSDU Storage Service.
Dataset metadata in a manifest is described by [the following schema..](https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/blob/master/Generated/manifest/GenericDataset.1.0.0.json)
A user needs to check Airflow log to validate what record ids from the manifest were processed and what record ids were not processed and the reason for rejection. Please keep in mind that your workflow status may be green (Successful), but some or all your resource ids were rejected.
To review the log, please navigate to your Airflow console and open osdu_ingest DAG:
<div>Image 19. Airflow. OSDU ingest DAG. Tree View</div>Then select your dag run from the drop-down, click on Go and click on the last step in the dag execution flow and click “View log”:
<div>Image 20. Airflow. OSDU ingest DAG. Graph View</div>Then click on Xcom and this screen will show you what ids were processed and what ids were skipped with the reason:
You can retrieve a metadata record from OSDU using Storage service that provides a basic functionality to retrieve a metadata record by its id or you can use Search service that provides very flexible and powerful mechanism to search for metadata.
OSDU uses Elasticsearch to provide search capabilities. After ingestion, metadata is indexed, and you can search this index. Please refer to Lucene query syntax for full documentation on how to search for any property used in the OSDU schemas:
Please note that for the same record id a record returned by the Storage service and Search service will look differently. This is because Storage service returns an original record and Search service returns an indexed record.
See below a couple of simple search queries.
#### Example: Search for a metadata record by id using Storage Service API
```http
GET https://{{STORAGE_HOST}}/records/{{record-id}}
```
#### Example: Search for a metadata record by id using Search Service API
```json
POSThttps://{{SEARCH_HOST}}/query
{
"kind":"*:*:*:*",
"limit":300,
"query":"id: \"{{record-id}}\""
}
```
### Download the original file
To get an original file, you may use similar services as were used for file ingestion.
First, you need to search for the needed metadata record of the dataset (see above). Then using dataset record id, you may need to request file DownloadUrl using File service, Dataset service or DDMSes.
#### Example: Get file DownloadUrl using File Service API
```http
GET {{FILE_HOST}}/files/{{record_id}}/downloadURL
```
After executing this request simply download the file using the provided link.
-[OSDU API Quick start demo](https://gitlab.opengroup.org/osdu/pmc/docs/-/blob/master/Google%20Cloud/Quick_Start_Guide_demo.mp4)
-[create OSDU on-prem using Helm chart](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-gcp-provisioning/-/blob/master/examples/simple_osdu_onprem/README.md)
-[create OSDU set of services within a single Google Cloud project](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-gcp-provisioning/-/blob/master/examples/simple_osdu/README.md)
-[create OSDU set of service within Azure](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/README.md)
