Skip to content

Ingestion updates

Type of change

  • Bug Fix
  • Feature

Does this introduce a change in the core logic?

  • [Yes]

Does this introduce a change in the cloud provider implementation, if so which cloud?

  • AWS
  • Azure
  • GCP
  • IBM

Updates description?

This MR comes with a batch of updates:


  • Validate entire manifest entity (GONRG-1783)
  • Surrogate keys replacement (GONRG-1652)
  • Auth logic uses Python SDK implementation (GONRG-1689)
  • New operator for manifest integrity (GONRG-1700)
  • New operator for schema validation (GONRG-1770)
  • Logic for ensuring Datasets, WPCs and WP referencial integrity (GONRG-1653)
  • Implementation of the batch uploading (GONRG-1650)
  • Added FileSource validation for Datasets (GONRG-1651)

Structure updates:

  • Removed obsolete dags (GONRG-1567)
  • has been updated (GONRG-1591)
  • Fix id composing (GONRG-1700)
  • Cleaned-up and renamed airflow variables (GONRG-1719)


  • Handle file variable fix
  • Fix download_as_bytes not supported in storage==1.13.2
  • Removed : (colon) symbol from the end of reference ids (GONRG-1911)

Merge request reports