This page captures various friction points between OSDU and OpenDES
- System of Record
- Metadata & master data visibility
- Concurrent Multi-Cloud
- Generic APIs vs Domain Specific APIs
- OSDU Release 1 APIs and OpenDES APIs
- Merging OSDU R1 and OpenDES Codebases
System of Record
- OSDU positions itself as a System of Record with a goal of removing direct access to data in source systems.
- OpenDES positions itself as the System of Reference for external data and System of Record for data that is created within the data ecosystems (derivatives).
While the above is academically at odds, from a practical perspective they may result in the same outcome. OpenDES focuses on capturing all data and improving it through the process of enrichment (improved discover-ability, quality, ....) to become the preferred source of information. Technically, it should address the goals of OSDU leaving the choice of removing data from source systems a choice of the Operator.
There is more to consider here. The principle we are referring to as System of Record here is based on the elimination of the previous convention that systems/apps that create/receive data items retain and act as the source forever. This convention leads to the complexities of point-to-point connectors to be able to reach into silo data sources. The OSDU intent is to bring stable data into OSDU by breaking the old convention. This not only removes the need for point-to-point connectors, but also avoids alternate sources being available, which can lead to parallel data derivation paths and the associated negative consequences.
There are clearly exceptions to this 'rule' for externally managed sets of data. There should however be no need to risk losing the benefits of true data integration and consolidation by leaving this as a choice point.
Determine what is necessary from a compliance perspective if OSDU is deemed a SoR
Metadata & master data visibility
- OSDU has taken the position that metadata and master data should be visible by default regardless of the entitlements of the data. The rationale behind this is that awareness is an enterprise concern.
- OpenDES has taken the position that entitlements should be uniformly applied. Thus, if access is not granted to data, then access should not be granted to its corresponding metadata.
- The treatment of master data can be addressed by assigning viewing privileges to the default data group.
- For metadata, we could use data security classification (Public, Private, Confidential, Secret) as a means to determine whether metadata should be generally searchable or not. We could make Public/Private metadata always visible and force Confidential/Secret metadata to honor the entitlements of the data.
Look to options that allow companies to configure metadata visibility separate from data visibility.
- OSDU has taken the position that individual customers will choose a single cloud provider to operate with and will working with a unified Identity model for both authentication and authorization.
- OpenDES was developed with a SaaS model in mind, and deals with applications and OpenDES instances that can exist on multiple clouds. Thus, it has to bridge identities among systems for authorizations even if authentication can be federated.
The complexity of bridging identities among multiple cloud providers will remain in DELFI rather than being pushed into OSDU.
Need to ensure consistency that one Cloud environment can talk to OSDU on another [Identity/Entitlement team]
Generic APIs vs Domain Specific APIs
- OSDU has prioritized data agnostic APIs to avoid introducing breaking changes in the API as new data types are added, or existing data types evolve. In time, OSDU is expected to add domain-specialized behavior through plug-in transformations within agnostic APIs and/or through specialized APIs. Some specialized APIs are expected to be deployed above the data platform, while others are deployed within the data platform.
- OpenDES has implemented data agnostic APIs but encourages the introduction of Domain specialized APIs to improve usability and performance. These Domain specialized APIs have to be governed to balance stability with evolution.
Both approaches are valid and can co-exist as long as Domain specific storage and APIs are treated as an enrichment of the Data and always derived from data that can be accessed via un-typed APIs.
OSDU Release 1 APIs and OpenDES APIs
- The initial Release 1 of OSDU has been implemented based on some of the original Shell Data Universe (SDU) APIs modified with agreement with key stakeholders in the OSDU Tiger Team.
- Documentation for the OSDU Release 1 APIs has not yet been developed making an effective comparison with OpenDES capabilities difficult.
- OpenDES has its own set of APIs which while broadly achieving the same purpose do so in a different way.
Two approaches can be taken to resolve this:
- Implement the current OSDU Release 1 APIs on the OpenDES code base
- Switch to the OpenDES APIs and do a gap analysis to identify any functional requirements missing from the OpenDES APIs expanding the functional footprint as required.
NOTE: To date, only two vendors have developed integration with the current OSDU APIs. Both of these vendors are open to updating their OSDU integration to support the OpenDES APIs and have committed to a review of the OpenDES APIs. Assuming the effort is acceptable to the relevant vendors, recommend option (2) above is selected. COMMENT: This consideration should only be taken up when the concept, classification, and behavior comparisons have been fully evaluated. Note also that the OSDU Release 1 API modifications are few and minor -- motivated by resolving some developmental issues to ease the burden for companies implementing Release 1. This should not be a significant impediment to the understanding and comparison activities.
- Capture detailed differences between existing OSDU and OpenDES APIs. Create the proposal. [Alan & Hrvoje]
- Formalize the design of system. [Stephen]
Merging OSDU R1 and OpenDES Codebases
- Both AWS and Microsoft have developed code bases to implement OSDU R1 these codebases are totally unrelated to OpenDES. Currently the AWS and Microsoft codebases are private proprietary repositories and are not Open Source (defined as publicly available repo with an open source license).
- OpenDES has its own codebase which is currently in the process of being released as Open Source with an Apache 2.0 license.
Two approaches can be taken to resolve this:
- Deprecate the current codebases for OSDU R1 and switch to the OpenDES codebase as shown in the diagram below:
- Create a hybrid codebase with code from the current OSDU R1 releases and the OpenDES codebase as shown in the diagram below:
NOTE: Neither of these approaches is intended to describe how the specifications of the APIs and the Data Definitions will be merged.
Start with approach 1 and opportunistically review bringing code from OSDU R1, but avoid building a Frankenstein system.