From f06ae1ae3eaa84ee01a507791962d26c38aa2976 Mon Sep 17 00:00:00 2001 From: Kishore Battula <Kishore.Battula@microsoft.com> Date: Thu, 11 Mar 2021 09:45:34 +0530 Subject: [PATCH] Added readme for azure --- README.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/README.md b/README.md index 6eec161..2b0b2b4 100644 --- a/README.md +++ b/README.md @@ -51,6 +51,23 @@ Environment dependencies might be installed by several ways: 1. Setting up an environment into the Cloud Composer Console. 2. Installing local Python library. Put your dependencies into *DAG_FOLDER/libs* directory. Airflow automatically adds *DAG_FOLDER* and *PLUGINS_FOLDER* to the *PATH*. +### Azure +To deploy the Ingestion DAGs to airflow, follow below steps. +- Identify the file share to which DAGs need to be copied. [This](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/infra/templates/osdu-r3-mvp/service_resources/airflow.tf#L71) is the file share where DAGs for airflow reside. +- Copy contents of */src/dags* to *airflowdags/dags* folder +- Copy contents of */src/plugins/hooks* to *airflowdags/plugins/hooks* +- Copy contents of */src/plugins/operators* to *airflowdags/plugins/operators* + +#### Installing Python Dependencies +Python dependencies can be specified as extra pip packages in airflow deployment [here](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/airflow/helm-config.yaml#L211) + +#### Environment Variables & Airflow Variables +Add variables manually in the Airflow UI or through airflow helm charts. [List of the required variables](#required-variables). + +Adding variables to helm charts can be found [here](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/airflow/helm-config.yaml#L157) + +More details about airflow variables can be found [here](https://airflow.apache.org/docs/apache-airflow/1.10.12/concepts.html?highlight=airflow_var#variables) + ## DAG Implementation Details -- GitLab