Explore projects

The File Service provides internal and external APIs to request for file location data, such as a signed URL per file upload. Using the signed URL, OSDU users will be able to upload their files for ingestion into the system.

Docs: https://osdu.pages.opengroup.org/platform/system/file

This CSV Parser DAG supports the ingestion workflow for CSV type files

Docker image for airflow deployment

Airflow DAG for transformation from SEGY to OpenZGY

OQM (Object to Queue) Mapper

Terraform module for deploying and managing OSDU services on Google Cloud Platform using Helm charts. Provides automated deployment and configuration of microservices in GKE (Google Kubernetes Engine) clusters.

The Ingestion Workflow service provides a wrapper functionality around the Apache Airflow functions and is designed to carry out preliminary work with files before running the Airflow Directed Acyclic Graphs (DAGs) that will perform actual ingestion of OSDU data.

Docs: https://osdu.pages.opengroup.org/platform/data-flow/ingestion/ingestion-workflow

Azure Infrastructure provisioning scripts and templates

This project is the home for required helm charts to install OSDU on Azure.

The project contains all Airflow dependencies need to OSDU.

Docs: https://osdu.pages.opengroup.org/platform/data-flow/ingestion/osdu-airflow-lib

Google Cloud Infrastructure provisioning scripts and templates.

Command-line interface for interacting with OSDU.

Airflow DAGs that consume Source Registry configuration to handle the fetch and ingest orchestration.

The Time-series Storage Core (or internal) Service is a final step in time-series ingestion pipeline. Processes published ingestion messages and stores data to the Time-series Storage.

The Manifest Ingestion DAG includes a Workflow Engine, an implementation of Apache Airflow, to orchestrate manifest/metadata ingestion via Storage Service