Dag Processing Design Descisions
- Strategy\Implementation to share data states between the airflow tasks.
- Strategy to clean up a shared state.
- Strategy to share the metadata across tasks.
- Strategy to deploy artifacts like jars.
- Ability to resuse an airflow step across dags.- , let’s say we have a task to convert the character set to UTF8… we need to make this clear that we should register this task/fragment once… and that it can be re-usable by all other DAG authors
- Custom operator to trigger dags within dags.