Dag Processing Design Descisions
-
Strategy\Implementation to share data states between the airflow tasks. -
Strategy to clean up a shared state. -
Strategy to share the metadata across tasks. -
Strategy to deploy artifacts like jars. -
Ability to resuse an airflow step across dags.- , let’s say we have a task to convert the character set to UTF8… we need to make this clear that we should register this task/fragment once… and that it can be re-usable by all other DAG authors -
Custom operator to trigger dags within dags.