Skip to content

Adding ingestion queue

Jose Jimenez Rocha requested to merge anjimene/addingIngestionQueue into master

All Submissions:

  • [YES] Have you added an explanation of what your changes do and why you'd like us to include them? Adding queue deployment to be used by ingestion workflow to get past scale problems with Airflow
  • [YES] I have updated the documentation accordingly.
  • [YES] My code follows the code style of this project.

Current Behavior or Linked Issues

Adding an optional queue deployment on the data storage account. By default the queue is not deployed. The ingestion workflow will use the queue to prevent problems with Airflow scalability

Does this introduce a breaking change?

  • NO

Merge request reports