Table of Contents
This document is work in progress
- Core Platform Services and Components
- Platform Architecture Overview
- Application Components and Technology Stack
- Cloud Provider Implementation
Core Platform Services and Components
The OSDU platform is a set of services and workflows orchestrated by an underlying cloud platform and deployed into your cloud environment of choice by infrastructure templates and deployment strategies provided by the CSP (Cloud Service Provider).
The OSDU also provide standard schemas and data types for upstream E&P data (as of this writing) with the goal of extending to other, newer energy data types. This diagram illustrates the core platform components, including services and standard data structures.
The platform exposes a common set of APIs with a standard access pattern, security and other entitlement behavior that is recognized and replicated across the platform. These services are specialized for certain interactions and designed to scale independently.
The core services can be orchestrated by the underlying platform and support CRUD operations and a common data lifecycle behavior. For example, Metadata flows into the system through the Storage service, which validates user access and entitlements to that data, which is done by querying the entitlement service. The search service is used to query and retrieve data based on indexed values, including attributes, coordinates, and other attributes. Indexing happens at the time of data ingestion, and the system requires valid schemas to index correctly. Data structures are validated by the schema service on write to ensure they are well structured, but invalid data can still flow in the system.
Indexing is done by the indexing service at the time of write, and indexes into an Elastic search instance setup by the CSP infrastructure. Elastic Search is the chosen indexing and search platform across all cloud providers because of its ability to index on coordinates and support polygon querying among other features.
This diagrams shows a strawman of the typical data flow, from external and existing data sources into the platform, and how applications can query and access the OSDU data: Simple Data Flow High Level Diagram
Extensibility and Lineage
The system supports history, and versioning of the data. <more on this, and discuss extensibility and lineage>
The system supports eventual consistency.
Platform Architecture Overview
The base platform is composed of a set of microservices illustrated in the diagram. These microservices are accessed through a standard API architecture published using Open API standards. User and/or service identity is used to authenticate the users accessing the APIs.
The core services are documented close to the source code to ensure alignment and freshness of the documentation.
A detailed inventory of the platform's core services and their APIs and documentation can be found on the following page : Core Microservices Catalog
Application Components and Technology Stack
The OSDU core services share a common set of core libraries stored in this core-lib repository. These include:
- A common cache client and cache dependency to Redis Cache.
- A common set of utility classes and Facades to manage partitioning (tenancy is OSDU).
- A set of common utilities such as JSON serialization, JWT, URL and other serialization tools.
- Common facades for OSDU services and their utilities, including Search, Legal, notification, partitioning etc..
- Common logging clients
- Common exception handling
- Common HTTP/HTTPS and REST client utilities
- Provider specific core dependencies, for AWS, Azure, Google and IBM
A couple of high level assertions about the core code:
- The code is mainly written in Java ((1.8 for now)
- Requires Maven to build
- Uses springboot as the universal hosting environment across all CSPs
- Each service has a complete list of dependencies in the provider implementation's POM.XML, including CSP specific and other service dependencies.
- All code has tests. Unit and integration tests use Junit (and sometimes cucumber)
Security, Entitlement and Compliance
Security of the system is a shared responsibility. The cloud platform, the provider, the platform operator, and the platform itself have their own responsibility to ensure end-to-end security of the OSDU.
The OSDU R3 has a provider specific entitlement service that manages user access to the data and services. The entitlement service per provider is developed as part of the code base and available as part of the core services. This services is an RBAC service and depends on user specific ACLs created and managed by the platform admin.
The legal services provides a simple rule engine to manage data governance and other data compliance rules.
There is currently an incubating service that aims to use policies to manage both user entitlements and compliance rules. This is in development but is not expected in the scope for OSDU R3 which can be accessed here
Other aspect of security involve security standards of the platform itself. These are captured as security requirements and documented and certified ahead of the release. For more information on the security parameters and the adherence, consult this document
Standard Logging Facade
Provider SPI Standards
Data Definition and Standards Documentation
Data Flow Components
The Ingestion and Enrichment processes in OSDU use an orchestration engine running on Airflow that enables component DAGs (directed Acyclic Graphs) developed and certified by the community. The OSDU provides a consistent, x-cloud workflow and portability and each cloud provider is responsible for deploying the framework (Airflow) and the DAGs as implemented and supporting platform level interaction of these ingestion components.
Cloud Provider Implementation
Technology Choice And Deployment Walkthrough
To provision your own Azure instance of OSDU, please refer to the reference architecture and templates described in this document
For the infrastructure template source code, browse through this repository
Elements of Azure's Infrastructure Architecture
The Azure application architecture for OSDU is based on microservices deployed into AKS. The technology stack and interactions are described here
Steps to onboard a new service on Azure are here