Feature change - Airflow seed data creation

Why is this change needed?

This change is needed because currently, we are not updating the status of the dag run in the workflow service. So, workflow service is not updated with the status of the dag run, if it was a success or failure.

Current behavior

When we trigger the workflow using the workflow service an entry is created in the workflow run collection with the status marked as Pending, but even on the completion of the dag run in the airflow the status in the workflow run collection remains unchanged.

Expected behavior

We want that as soon as the dag run completes on the airflow the status should be updated in the workflow run collection as well.

Current Design

Workflow run API in the workflow service calls the airflow API to trigger the dag with the given dag id. After which the airflow schedules the run.

Initial Design Proposal

We are proposing to encapsulate the dag run behind a controller dag. This controller dag uses a custom dag run operator to trigger the child dag run. This controller dag also uses a custom dag status sensor which checks for the child dag run to complete periodically, on completion of the dag run this sensor updates the status in the workflow run collection. This merge request contains the seed files i.e, the controller dag, custom dag run operator and the custom dag status sensor. Controller dag needs to be uploaded to file share and along with it the operator and the sensor also need to be uploaded.