Data Ingestion issueshttps://community.opengroup.org/groups/osdu/platform/data-flow/ingestion/-/issues2022-09-15T23:49:37Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/issues/47Documentation: Best practices for ingestion DAGs2022-09-15T23:49:37ZAlan HensonDocumentation: Best practices for ingestion DAGsWe need a guide or document that offers best practices recommendations to those constructing ingestion (or enrichment) related DAGs. This document should cover things such as:
- DAG Operator composability recommendations
- Performance c...We need a guide or document that offers best practices recommendations to those constructing ingestion (or enrichment) related DAGs. This document should cover things such as:
- DAG Operator composability recommendations
- Performance considerations
- Recommended property use (see https://community.opengroup.org/osdu/documentation/-/issues/80)
- Othershttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/25Documentation: Energistics/WITSML User Guide2022-09-15T23:49:37ZAlan HensonDocumentation: Energistics/WITSML User GuideCreate the user guide for the Energistics/WITSML Ingestion workflow.Create the user guide for the Energistics/WITSML Ingestion workflow.https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/issues/26Documentation: Manifest Ingestion User Guide2022-09-15T23:49:36ZAlan HensonDocumentation: Manifest Ingestion User Guidehttps://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/18Ingestion Framework Guiding Principles2023-03-09T18:15:53ZStephen Whitley (Invited Expert)Ingestion Framework Guiding Principles
## Ingestion Framework Guiding Principles
These guiding principles are shared by all the SLB authored user stories reflecting the capabilities of OpenDES
* Simple, Dedicated and Efficient APIs as ingestion entry points for each data f...
## Ingestion Framework Guiding Principles
These guiding principles are shared by all the SLB authored user stories reflecting the capabilities of OpenDES
* Simple, Dedicated and Efficient APIs as ingestion entry points for each data format
* Store original high-fidelity data as is. Data in its original form must land first in the most appropriate store. Ex: - A DLIS file must land in the File DMS, A ZGY seismic survey must land in the Seismic DMS, etc. Once the data, in its original form, lands in the appropriate DMS, Parsers/Scanners/Enrichment processes can be applied to that data. The original data is stored as-is and parsers/scanner/clean-up processes will output derived entities.
* Framework should inherently enforce data lifecycle, Original ---> Well Known Structure ---> Well known Entity
* Extensibility of Parsers/Scanner/Enrichment processes through Configurations/Registrations
* Track flow and lifecycle of data in the data platformM1 - Release 0.1https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/18R3 parser documentation2022-09-15T23:49:37ZJana ScheyR3 parser documentationSample issueSample issueJana ScheyJana Schey2020-12-11