ADR: Ingestion Service - R3 Schema support

Context

  • Data Loading requirements to enforce validation of users request before pass control to DAG logic.
  • Currently all validation logic implemented in DAG itself.
  • It was decided to do this work post-R3, but still will keep Ingestion Service in R3 architecture as an entry point for Ingestion.
  • To work with R3, Ingestion Service endpoints should be updated to support R3 schemas /submitWithManifest.

Scope

  • Add a new endpoint /api/v2/submit/manifest that will support R3 Manifest format.
  • Update Ingestion Service logic to work with updated API of Workflow service (see ADR ???)
  • See OpenAPI Spec for details

Decision

  • Introduce a new endpoint
  • Code change in Ingestion Service > Core to work with updated API of Workflow service

Rationale

  • Required change for E2E R3 Ingestion

Consequences

  • Work will be done by GCP
  • Most of the changes will be done is Ingestion Service > Core
  • No CSPs involvement required as there is not SPI layer in Ingestion service

When to revisit

  • Post R3