... | ... | @@ -8,10 +8,10 @@ This page is an organic page that will grow over time. This page's intent is to |
|
|
At the time of writing, [Apache Airflow](https://airflow.apache.org/) 1.10.x is the designated workflow engine for OSDU. Airflow provides DAG authors the ability to write DAGs once and have them run on any OSDU implementation. This section addresses some of the steps you can take to optimize Airflow. Airflow also ships with recommendations on how to optimize - see [here](https://airflow.apache.org/docs/apache-airflow/stable/faq.html#how-can-my-airflow-dag-run-faster).
|
|
|
|
|
|
- If possible, consider upgrading to Airflow 2.0. Airflow 2.0 addresses some of the lessons learned from Airflow 1.x, such as allowing multiple schedulers which enable high availability.
|
|
|
- Configure your Airflow infrastructure to remain running at all times (vs. spinning down to zero Airflow instances if no jobs are running). This will minimize the initial time hit when a new request comes in as the Airflow infrastructure spins up. _More detail coming_
|
|
|
- Increase the configured payload size for the webserver (see the `dag_run` table) _More detail coming_
|
|
|
- Increase the max Xcom size by changing out the backend database. See [here](https://marclamberti.com/blog/airflow-xcom/) for additional XCom options.
|
|
|
- Increase the `max_active_runs_per_dag` parameter (see more details [here](https://airflow.apache.org/docs/apache-airflow/stable/faq.html#how-can-my-airflow-dag-run-faster))
|
|
|
- Configure your Airflow infrastructure to remain running at all times (vs. spinning down to zero Airflow instances if no jobs are running). This will minimize the initial time hit when a new request comes in as the Airflow infrastructure spins up.
|
|
|
- Increase the configured payload size for the webserver (see the `dag_run` table)
|
|
|
- Increase the max Xcom size by changing out the backend database or reconfiguring the size of the column in the Airflow database where serialization occurs. See [here](https://marclamberti.com/blog/airflow-xcom/) for additional XCom options.
|
|
|
- Increase the `max_active_runs_per_dag` parameter (see more details [here](https://airflow.apache.org/docs/apache-airflow/stable/faq.html#how-can-my-airflow-dag-run-faster)) and also the `dag_concurrency` parameters (see more details [here](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-concurrency)).
|
|
|
|
|
|
## DAG Development ##
|
|
|
|
... | ... | |