OpenDES Ingestion Service Provider User Stories
These stories come the data workflow team and representing the current capabilities available in OpenDES
Stories
-
As a 3rd party service provider, I need to be able to ingest data directly into an appropriate DDMS (e.g. Object DDMS, Wellbore DDMS, etc) directly using ETL tools or my own proprietary software. -
As a 3rd party service provider, I need the ability to register in my proprietary parsers into the framework so that I can ingest data that other parsers can’t – for example a parser for UKOOA P7/2000 trajectory data exchange format. -
As a 3rd party service provider, I need the ability to register my proprietary enrichment routines to enrich data in ways that other enrichment routines can’t – for example using machine learning models to suggest infills for missing values. -
As a 3rd party service provider, I need the ability to continuously load data in the platform by building agents for various data sources, applications, repositories.
Supporting Detail
These guiding principles apply to all of the user stories that support OpenDES behavior
- Simple, Dedicated and Efficient APIs as ingestion entry points for each data format
- Store original high-fidelity data as is. Data in its original form must land first in the most appropriate store. Ex: - A DLIS file must land in the File DMS, A ZGY seismic survey must land in the Seismic DMS, etc. Once the data, in its original form, lands in the appropriate DMS, Parsers/Scanners/Enrichment processes can be applied to that data. The original data is stored as-is and parsers/scanner/clean-up processes will output derived entities.
- Framework should inherently enforce data lifecycle, Original Well Known Structure Well known Entity
- Extensibility of Parsers/Scanner/Enrichment processes through Configurations/Registrations
- Track flow and lifecycle of data in the data platform
Acceptance Criteria
Ability of a service provider to use and extend the ingestion and enrichment services to bring data into OSDU.