Comparing OSDU & OpenDES Schema Semantics

This page is authored to help compare the OSDU and OpenDES models, with the aim to:

Gain a collective understanding of both the worlds, collect feedback
Help Architects & Data Modelers review possible strategies to blend the two models

This page & the recommendations are collectively authored by the following members, during the three week period from 30 Sep 2019 - 18 Oct 2019.

Alan C Doniger (Shell)
Alex Narayanan (SLB)
Ethiraj Krishnamanaidu (SLB)
Hrvoje Markovic (SLB)
Thomas Gehrmann (SLB)
Vincent J Kowalski (Shell)

Update on 13-JAN-2020 The proposals were submitted & reviewed by core members of EA subcommittee in December 2019. The original proposals have remained more or less unchanged. The summary can be found [here].(https://community.opengroup.org/osdu/documentation/-/wikis/Resolutions-for-GroupType,-WorkProduct-and-Kind-identity-discussions) Subsequently this page is updated to reflect those additional recommendations.

Kind vs. Master Data, Reference Data, Work Product & Work Product Components - Added more examples, linked the page that captures the EA notes.

Handling Identifiers - updated rationale to link the parent page.

Kind vs. Master Data, Reference Data, Work Product & Work Product Components

Problem Statement

OpenDES and OSDU have different ways to identify a schema - Kind vs ResourceType.

OpenDES

Each record in OpenDES storage has a kind property, identifying to which schema it wants to be associated with.

The pattern currently used is <namespace>:<source>:<entityType>:<versionMajor>.<versionMiddle>.<versionMinor>, example common:boem:well:1.0.0.

namespace : the authority who published this schema and manages versions to it, eg. slb
source: source of this data, such as ihs, boem, wke, ref-data etc. The latter two may have a specific meaning such that wke relates to master/curated data and ref-data refers to reference values & types such as log codes.
entityType: name of the entity, wells, logs, seismic3d etc.
<versionMajor>.<versionMiddle>.<versionMinor> : Semantic versioning scheme for the schema. ??Include a reference to schema service, on implementation & guidelines on-increment a version??

OSDU

OSDU has an overarching concept of Master data, Work Product, Work Product Components, File and Reference Data. And it is within this hierarchy, the "kind" is embedded.

Each of these same concepts follow a "ResourceTypeID" to identify its type. The ResourceTypeID is of the format srn:<namespace>:type:<GroupType>/<IndividualType>:<version>, wherein:

namespace: the authority who published this schema and manages versions to it, eg. osdu.
GroupType: Can be one of master-data, reference data, work-product, work-product-component or file
IndividualType: The name of the specific type classification within a group type - example, Field, WellLogWorkProduct etc.
version: The version of of the resource type - in the format [0-9]+

Proposal

We propose that the following model that emulates the best of OSDU & ODES

OSDU	ODES	Proposal	Comments
namespace	Namespace	Namespace	The authority which published and owns the schema, such as shell, cvx, osdu
NONE	source	Source	The origin of schema that could be come from a.) Formats defined by solutions such as Petrel, ProSource, IHS, OpenWorks (or) b.) Industry standard formats such as SEGY-Rev1, LAS etc
GroupType	NONE	Add GroupType as a Property inside the JSON schema - with values matching with the OSDU GroupType - Master, WP, WPC and Reference Data
IndividualType	EntityType	Type	--
Version [0-9]+	Version (Semantic Version - n.n.n)	Version (Semantic Version - n.n.n)	Adapt semantic version

You'll notice that GroupType is retained and carried forward from OSDU.

Below is the illustration of the current OSDU model merging into the combined model

Below are some of the examples on how the Kind would look across different domains.

Work Products named Project and Study

   * opengroup:osdu:project:1.0.0
   * opengroup:osdu:study:1.0.0
   * shell:osdu:project:1.0.0
   * shell:osdu:study:1.0.0
   * schlumberger:osdu:project:1.0.0
   * schlumberger:osdu:study:1.0.0

Work Product Components named Well Logic and Survey

  * opengroup:osdu:welllog:1.0.0
  * opengroup:osdu:survey:1.0.0
  * shell:osdu:welllog:1.0.0
  * shell:osdu:survey:1.0.0
  * schlumberger:osdu:welllog:1.0.0
  * schlumberger:osdu:survey:1.0.0

Master Data named Well and Field

  * opengroup:osdu:well:1.0.0
  * opengroup:osdu:field:1.0.0
  * shell:osdu:well:1.0.0
  * shell:osdu:field:1.0.0
  * schlumberger:osdu:well:1.0.0
  * schlumberger:osdu:field:1.0.0

Well log example

opengroup:osdu:well:1.0.0 - (groupType=Master)
opengroup:osdu:welllog-channelset:1.0.0 – (groupType=WorkProductCompoent)
opengroup:osdu:project:1.0.0 or opengroup:osdu:study:1.0.0 – groupType=Master)
opengroup:osdu:File:1.0.0 (groupType == File).

Rationale

Please see https://community.opengroup.org/osdu/documentation/-/wikis/Resolutions-for-GroupType,-WorkProduct-and-Kind-identity-discussions for additional rationale.

Use of $ref & $allOf, to reuse schema definitions

Problem Statement

OpenDES does not refer to other definitions using $ref and $allOf; the current search-focused schema definition in the Storage service does not have the expressive power. Schlumberger is working on a new schema service, which promotes composition based on the JSON schema standard draft # 7 in a scripted and automated way, which is also included in the Open API Specification 3 (OAS3). The latter supports the use of JSON schema keywords allOf, which is used in OSDU to derive a concrete entity type. Once the new schema service is deployed, OpenDES will be able to support OSDU's pattern of abstract entity types.

Proposal

Adopt the use of references in OSDU as a means to make them precise & reusable definitions. The existing #definitions in OpenDES will change to adopt this guideline and will also confirm with JSON Schema Draft #7 (closed) and OAS3.
Schema definitions retrieved from this system may be self-contained, i.e. $ref only refers to internal model definitions #/definitions/<theElementReferredTo>.

Handling Identifiers

Problem Statement

Following semantic differences are noted between the two systems in the way ID is generated.

In OSDU, identifier takes the form srn:<GroupType>/<IndividualType>:[^:]+:[0-9]*, for example srn:master-data/Basin:MyBasin:1
In OpenDES, the identify takes the form <DataPartitionID>:<Source>:<GUID>, for example, slb:ihs:53ec94f2-4884-5c2a-8142-e603b0919955:1543614413503417

Proposal

The condition for this proposal is the flexibility to change the OpenDES DataPartitionID with srn. [Hrvoje to confirm]

Timeline R2:

Adopt srn for entity types covered by OSDU, which correspond to well-known schemas only.
Keep OpenDES identity for raw records and non-OSDU entity types.

Timeline R3:

Adopt URNs instead of SRNs
Real adoption of URN standard requires the registration of a namespace with W3C. This should be done in a timely fashion and require data migration. The result, however, would be industry standard.
Introduce the concept of a data partition into the URN, which may be an OSDU instance.

Structure of an id (aka OSDU ResourceID):

urn:<namespace id>:<type>:<instance>:<version>

namespace id composed of namespace and optional data partition id
type is the type name as appears in kind
instance and version are defining the identity

For example,

urn:osdu.opengroup.org/US:Well:W101:1276423
urn:www.bigoil.com/US:well:MyWell001:42388463
`urn:www.bigoil.com:well:W102:289546 (data partition is omitted here)

The <instance>:<version> can be used a unique identity to look up data on a 'local' instance, such that in cases where the fully resolved ID is not present, the systems can make the next best choice. This recommendation is added to help migrate data which may not have the fully resolved syntax for ID.

Rationale

Please see https://community.opengroup.org/osdu/documentation/-/wikis/Resolutions-for-GroupType,-WorkProduct-and-Kind-identity-discussions for additional rationale.

Staged introduction of new id scheme with limited data migration.
Self-documenting id
Standards compliance.
The URNs can be used to validate relationships by defining a regular expression pattern for the id in the schema.
Adoption of the data-partition.
Consistent with the Kind representation as recommended in the earlier sections

Frame of Reference Handling

Problem Statement

OpenDES annotates certain attributes with a "Frame of Reference", to help contextualize the value it carries. A few examples here are:

coordinate reference systems (CRS)
measurements
units of measure
date and date-time

OSDU has the concept, it is not collectively referred to as frame of reference.

persistableReference Annotation

annotations are called as "perstitableReference". The values encoded under the reference are kept fully self-contained with they data they describe. Each reference value is a stringified JSON structure fully describing the reference. The schema is for each of the types above is provided, see JSON schema persistableReference, or in this documentation.

Persistable Reference Examples

Coordinate Reference System Example (same example as in section Geospatial Definitions:

"{\"lateBoundCRS\":{\"wkt\":\"PROJCS[\\\"NAD_1927_StatePlane_Louisiana_South_FIPS_1702\\\",GEOGCS[\\\"GCS_North_American_1927\\\",DATUM[\\\"D_North_American_1927\\\",SPHEROID[\\\"Clarke_1866\\\",6378206.4,294.9786982]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],PROJECTION[\\\"Lambert_Conformal_Conic\\\"],PARAMETER[\\\"False_Easting\\\",2000000.0],PARAMETER[\\\"False_Northing\\\",0.0],PARAMETER[\\\"Central_Meridian\\\",-91.3333333333333],PARAMETER[\\\"Standard_Parallel_1\\\",29.3],PARAMETER[\\\"Standard_Parallel_2\\\",30.7],PARAMETER[\\\"Latitude_Of_Origin\\\",28.6666666666667],UNIT[\\\"Foot_US\\\",0.304800609601219],AUTHORITY[\\\"EPSG\\\",26782]]\",\"ver\":\"PE_10_3_1\",\"name\":\"NAD_1927_StatePlane_Louisiana_South_FIPS_1702\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"26782\"},\"type\":\"LBC\"},\"singleCT\":{\"wkt\":\"GEOGTRAN[\\\"NAD_1927_To_WGS_1984_79_CONUS\\\",GEOGCS[\\\"GCS_North_American_1927\\\",DATUM[\\\"D_North_American_1927\\\",SPHEROID[\\\"Clarke_1866\\\",6378206.4,294.9786982]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],METHOD[\\\"NADCON\\\"],PARAMETER[\\\"Dataset_conus\\\",0.0],AUTHORITY[\\\"EPSG\\\",15851]]\",\"ver\":\"PE_10_3_1\",\"name\":\"NAD_1927_To_WGS_1984_79_CONUS\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"15851\"},\"type\":\"ST\"},\"ver\":\"PE_10_3_1\",\"name\":\"NAD27 * OGP-Usa Conus / Louisiana South [26782,15851]\",\"authCode\":{\"auth\":\"SLB\",\"code\":\"26782079\"},\"type\":\"EBC\"}"

Unit of Measure Example:

"{\"abcd\":{\"a\":2298.35,\"b\":5.0,\"c\":9.0,\"d\":0.0},\"symbol\":\"degF\",\"baseMeasurement\":{\"ancestry\":\"K\",\"type\":\"UM\"},\"type\":\"UAD\"}"

Proposal

We recommend to include the concept of "persistableReference" as the means to describe such values tha may need a frame of reference to describe the data in a consistent and correct manner.

Relationship Handling

Problem Statement

On the surface OSDU and OpenDES relationship handling is done differently.

OpenDES

In OpenDES the recommended pattern is to register the related entities in a relationships block. The latter is a dictionary where the key is the name of the relationship. The value is a nested structure containing the natural key and/or the exact id of the related entity. This allows the registration of natural keys in case the target entity is not yet present; subsequent enrichment can then populate the id once the target entity has been created. It is also possible to specify the exact version number if so required. OpenDES uses the schema fragment toOneRelationship for this purpose, definition here.

In the JSON schema the target entity types can be specified (example here), which is equivalent to the name of the target entity type in the OSDU srn.

For low cardinality relationships it is possible to declare toManyRelationship, which consists of an array of natural key/id/version identifiers, definition here. So far toManyRelationship is rarely in use; typically child instances refer to parents as toOneRelationship

Note that the diagram above shows the desired model (today's implementation is slightly different).

High cardinality relationships are expected to be expressed via associations, i.e. a separate entity, referring to both the source and target entity. The association's relationships are defined in an ordinary relationships block, see schema here. An example record associating a wellbore to a collection is shown here:

Note: Modifying relationships inside the relationships block change the entity and cause a new entity version. Changing associations do not affect the source and target entities to be associated. The behavioral difference is important to recognize during schema design.

Proposal

After careful analysis, we concluded two approaches are in general alignment (i.e., this is not a friction point). The OpenDES approach is an optional recommendation.

Modelling Geospatial Information

Problem Statement

OSDU provides reference data that allows for the modeling of 2D Spatial shapes. OpenDES is GeoJSON-based. GeoJSON provides capabilities for both 2D and 3D shapes. The fricion-point, if any, is that each system provides different spatial capabilities.

Variants

The well-known schema definitions promote two representations, which are capable of holding simple geometric shapes:

GeoJSON FeatureCollection (as per default referring to WGS 84). This representation is preferred for discovery. Usually, the data are derived from geospatial data in the original, native coordinate reference system context, see next bullet.
AnyCrsFeatureCollection is similar to the GeoJSON structure but cannot be confused with GeoJSON because all structure type codes are prefixed with AnyCrs. The geospatial context is explicitly given by an additional mandatory property (persistableReferenceCrs); for 3D coordinates the unit for the 3rd coordinate must be defined (persistableReferenceUnitZ). This representation is used to capture the original geospatial data in their original context without any numerical manipulation.
Legacy geo-point - A lot of early schemas referred to a kind core:dl:geopoint:1.0.0, which is used in indexing.

The CRS-conversion service supports changing the frame of reference for such geospatial representations, specifically from AnyCrsFeatureCollection to WGS 84 as GeoJSON FeatureCollection.

Geospatial Definitions

Definitions contain Esri well-known text (WKT); although not ISO WKT compliant, Esri WKTs are widely used in the industry, (e.g. ArcGIS). For automation, the use of early-bound CRSs are promoted. This means a CRS based on a local datum is associated with a transformation to the 'hub CRS' WGS 84. Under appropriate conditions, data are convertible from one local CRS via the 'hub CRS' to another local CRS.

Example early-bound CRS

Below EPSG:26782 bound to NADCON continental US transformation EPSG:15851. The JSON structure is usually stringified into a single, long string and associated with the data:

    {
        "authCode": {
          "auth": "SLB",
          "code": "26782079"
        },
        "type": "EBC",
        "ver": "PE_10_3_1",
        "name": "NAD27 * OGP-Usa Conus / Louisiana South [26782,15851]",
        "lateBoundCRS": {
          "authCode": {
            "auth": "EPSG",
            "code": "26782"
          },
          "type": "LBC",
          "ver": "PE_10_3_1",
          "name": "NAD_1927_StatePlane_Louisiana_South_FIPS_1702",
          "wkt": "PROJCS[\"NAD_1927_StatePlane_Louisiana_South_FIPS_1702\",GEOGCS[\"GCS_North_American_1927\",DATUM[\"D_North_American_1927\",SPHEROID[\"Clarke_1866\",6378206.4,294.9786982]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],PROJECTION[\"Lambert_Conformal_Conic\"],PARAMETER[\"False_Easting\",2000000.0],PARAMETER[\"False_Northing\",0.0],PARAMETER[\"Central_Meridian\",-91.3333333333333],PARAMETER[\"Standard_Parallel_1\",29.3],PARAMETER[\"Standard_Parallel_2\",30.7],PARAMETER[\"Latitude_Of_Origin\",28.6666666666667],UNIT[\"Foot_US\",0.304800609601219],AUTHORITY[\"EPSG\",26782]]"
        },
        "singleCT": {
          "authCode": {
            "auth": "EPSG",
            "code": "15851"
          },
          "type": "ST",
          "ver": "PE_10_3_1",
          "name": "NAD_1927_To_WGS_1984_79_CONUS",
          "wkt": "GEOGTRAN[\"NAD_1927_To_WGS_1984_79_CONUS\",GEOGCS[\"GCS_North_American_1927\",DATUM[\"D_North_American_1927\",SPHEROID[\"Clarke_1866\",6378206.4,294.9786982]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],GEOGCS[\"GCS_WGS_1984\",DATUM[\"D_WGS_1984\",SPHEROID[\"WGS_1984\",6378137.0,298.257223563]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],METHOD[\"NADCON\"],PARAMETER[\"Dataset_conus\",0.0],AUTHORITY[\"EPSG\",15851]]"
        }
      }

Proposal

Although the OSDU and OpenDES are different representations of spatial data, GeoJSON is a superset of that which is expressed in OSDU. As such, we do not see an inherent conflict here and would recommend that OSDU adopt the richer set of capabilities available in GeoJSON (i.e., OpenDES).

General Structure of the JSON Schema

Problem Statement

As shown below the properties have little overlap, except ResourceTypeID, which corresponds to kind and ResourceID, which corresponds to id.

OSDU

ResourceTypeID - "The SRN of the resource's resource type."
ResourceID - "The SRN which identifies this OSDU resource object at the version level."
ResourceHomeRegionID - "The name of the home [cloud environment] region for this OSDU resource object."
ResourceHostRegionIDs - "The name of the host [cloud environment] region(s) for this OSDU resource object."
`ResourceObjectCreationDateTime" - "Timestamp of the time at which Version 1 of this OSDU resource object was originated."
ResourceVersionCreationDateTime - "Timestamp of the time when the current version of this resource entered the OSDU."
ResourceCurationStatus - "Describes the current Curation status. Possible values - CREATED, CURATING, CURATED."
ResourceLifecycleStatus - "Describes the current Resource Lifecycle status. Possible values - LOADING, RECEIVED, ACCEPTED, RESCINDED, DELETED,"
ResourceSecurityClassification - "Classifies the security level of the resources. Possible values = RESTRICTED, CLASSIFIED, CONFIDENTIAL, MOST-CONFIDENTIAL"

OpenDES

All instances have the same basic structure with properties

id - unique identifier of this instance across all versions
kind - schema identifier/schema version
acl - access control
legal - legal tags and relevant countries
version - system assigned version identifier; not required on POST
meta[] - Frame of Reference definition(s); not required on POST
data - the data block typically specific to the kind schema

OpenDES uses self-contained references for units, measurements, spatial references (CRS, datum transformations), time(-zone) references. Further details in this branch.

Other reference data in the OSDU sense are intended to be defined via ordinary catalog and catalogMember type records. Ideally the schema should refer to such catalogs to constrain values for certain properties. This is not yet in use.

Proposal

Adopt id for ResourceID
Adopt kind for ResourceTypeID.
Add all other OSDU properties as optional, except
1. ResourceObjectCreationDateTime and
2. ResourceVersionCreationDateTime
OpenDES Storage to set ResourceObjectCreationDateTime and ResourceVersionCreationDateTime on PUT.
Make dates and ResourceLifecycleStatus searchable.

Rationale

This is to minimize data migration effort while maintaining as much of the common semantics as possible.

Modelling for Versions

Problem Statement

At the first glance the versionsing schemes appear different.

OpenDES

The version in OpenDES is an auto-generated long integer and separate from the id. The numbers do not reveal any sequence.

OSDU

The version in OSDU is part of the SRN and it is sequential. The SRN is formed using a defined pattern so that the version number can be extracted.

Proposal

In essence the versioning schemes are equivalent. Thus there isn't a need for reconciliation. The OpenDES versioning scheme is recommended. The end-user usability should however be considered; it would be preferable to expose the sequence of (long integer) version numbers as simple increasing sequence.

OSDU Wellbore example schema

Abridged wellbore schema using added id and kind and $ref kind reference notation (R2 scope)

Comparing OSDU & OpenDES Schema Semantics

Contents

Kind vs. Master Data, Reference Data, Work Product & Work Product Components

Problem Statement

OpenDES

OSDU

Proposal

Rationale

Use of $ref & $allOf, to reuse schema definitions

Problem Statement

Proposal

Handling Identifiers

Problem Statement

Proposal

Rationale

Frame of Reference Handling

Problem Statement

persistableReference Annotation

Proposal

Relationship Handling

Problem Statement

OpenDES

Proposal

Modelling Geospatial Information

Problem Statement

Variants

Geospatial Definitions

Proposal

General Structure of the JSON Schema

Problem Statement

OSDU

OpenDES

Proposal

Rationale

Modelling for Versions

Problem Statement

OpenDES

OSDU

Proposal

OSDU Wellbore example schema