removing with_validation boolean flag (!39) · Merge requests · Open Subsurface Data Universe Software / Platform / Data Flow / Data Ingestion / Ingestion DAGs

Closed Brady Spiva [AWS] requested to merge fix/redundant-validations into master Apr 14, 2021

Type of change

Bug Fix
Feature

Does this introduce a change in the core logic?

[Yes]

Does this introduce a change in the cloud provider implementation, if so which cloud?

AWS
Azure
GCP
IBM

Updates description?

This addresses issue #54 (closed), removing redundant validation steps to speed up manifest parsing. The manifest parser DAG performs validations for referential integrity and schema conformity in separate operators, so there is no need to repeat the validations here in the manifest parsing stage of this DAG.