Ingestion reference data validation
- Under review
This was approved in January OSDU R2 Decisions review and has since been revisited.
Context & Scope
- Currently we are only validating manifests, not reference targets; we should validate that reference data to which an SRN refers exists, as well as other relationships
- How should this work? OpenDES model is to load and cleanup after based on batch scan checks; OSDU philosophy tends to be to only allow valid data in; if they’re doing as part of pre-processing, then capture as long-term backlog issue.
- Gap Fit: Understand how validation is done today (ex, is it well formed JSON)
- Create an ADO story
- Dependency on Schema Service developed by SLB Pune Team
- Ethiraj to update on readiness
- Cross-Cloud Testing and Validation
When to revisit
Tradeoff Analysis - Input to decision
Alternatives and implications
Propose postponing post-R2; this implies that bad data can be loaded.
Decision criteria and tradeoffs
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information