Adding ingestion queue
All Submissions:
- [YES] Have you added an explanation of what your changes do and why you'd like us to include them? Adding queue deployment to be used by ingestion workflow to get past scale problems with Airflow
- [YES] I have updated the documentation accordingly.
- [YES] My code follows the code style of this project.
Current Behavior or Linked Issues
Adding an optional queue deployment on the data storage account. By default the queue is not deployed. The ingestion workflow will use the queue to prevent problems with Airflow scalability
Does this introduce a breaking change?
- NO