Ingestion Framework Guiding Principles
Ingestion Framework Guiding Principles
These guiding principles are shared by all the SLB authored user stories reflecting the capabilities of OpenDES
- Simple, Dedicated and Efficient APIs as ingestion entry points for each data format
- Store original high-fidelity data as is. Data in its original form must land first in the most appropriate store. Ex: - A DLIS file must land in the File DMS, A ZGY seismic survey must land in the Seismic DMS, etc. Once the data, in its original form, lands in the appropriate DMS, Parsers/Scanners/Enrichment processes can be applied to that data. The original data is stored as-is and parsers/scanner/clean-up processes will output derived entities.
- Framework should inherently enforce data lifecycle, Original ---> Well Known Structure ---> Well known Entity
- Extensibility of Parsers/Scanner/Enrichment processes through Configurations/Registrations
- Track flow and lifecycle of data in the data platform