ADR : New Airflow operator (and respective DAG) to get information about Airflow environment
Context:
If Airflow environments are created not with fully managed services (e.g., Google Composer or Amazon MWAA) it is difficult to check what versions of osdu-airflow-lib
, osdu-api
, osdu-ingestion
and other libs are used. For the instance, if Airflow is deployed in K8S the only way to check the versions is to go into running pods and call pip list
.
Also, Airflow variables
might be set as environmental variables, and they don't appear in Web UI that makes it hard to debug.
Proposal:
The proposal is to create a new Airflow operator and the DAG that will return the environmental information of Airflow instances.
- The DAG should be triggered with a system workflow (ADR: ingestion-workflow#118 (closed))
- The operator should return Airflow variables and the versions of
osdu-*
libs.