Skip to content

GitLab

Added iaap auth for airflow stable api (GONRG-3753)

Merged Rustam Lotsmanenko (EPAM) requested to merge gcp-iaap-auth-airflow-v2 into master

Description:

Refactor workflow to use different approaches for auth with Airflow 2.0. This MR brings Airflow 2.0 stable API to workflow service, but it has hardcoded Basic auth as the only way to authenticate requests to Airflow. We need to un-hardcode authentication and verify work with Airflow 2.0 stable API.

Airflow2 can be deployed either in Composer or K8S.

If it is deployed in Composer, then Authorization remains the same as it was before (Bearer: <IAAP_Token>), because Composer uses its own auth backend for Stable API
If Airflow is deployed in K8S, then we can use Basic Auth.

Workflow Service must have the possibility to choose one of these two options depending on where AF2 is deployed.

In order to switch between stable and experimental API env variable OSDU_AIRFLOW_VERSION2 may be used.
In order to switch between basic and IAAP auth env variable AIRFLOW_IAAP_MODE may be used.

How to test:

Functionality was tested with experimental and stable API

Changes include:

  • Refactor (a non-breaking change that improves code maintainability).
  • Bugfix (a non-breaking change that solves an issue).
  • New feature (a non-breaking change that adds functionality).
  • Breaking change (a change that is not backward-compatible and/or changes current functionality).

Changes in:

  • GCP
  • Azure
  • AWS
  • IBM
  • Common code

Dev Checklist:

  • Added Unit Tests, wherever applicable.
  • Updated the Readme, if applicable.
  • Existing Tests pass
  • Verified functionality locally
  • Self Reviewed my code for formatting and complex business logic.
Edited by Rustam Lotsmanenko (EPAM)