OSDU Airflow Library
OSDU Airflow Library is the package providing Airflow specific logic.
Contents
Getting Started
Required Airflow Variables
The following variables are used by osdu_api clients. If some of them are missing osdu_api will try to find corresponding values in osdu_api.ini file, which path is set in OSDU_API_CONFIG_INI environmental variable.
| Variable name | Example |
|---|---|
| core__service__dataset__url | <http||https>://\<service-host>/api/dataset/v1 |
| core__service__file__host | <http||https>://\<service-host>/api/file |
| core__service__file__url | <http||https>://\<service-host>/api/file/v2 |
| core__service__partition__url | <http||https>://\<service-host>/api/partition/v1/ |
| core__service__schema__host | <service-host> |
| core__service__schema__url | <http||https>://\<service-host>/api/schema-service/v1 |
| core__service__search__host | <service-host> |
| core__service__search__url | <http||https>://\<service-host>/api/search/v2 |
| core__service__seismic__url | <http||https>://\<service-host>/api/seismic-store/v3 |
| core__service__storage__host | <service-host> |
| core__service__storage__url | <http||https>://\<service-host>/api/storage/v2 |
| core__service__unit__url | <http||https>://\<service-host>/api/unit/v2 |
| core__service__workflow__host | <http||https>://\<service-host>/api/workflow |
| core__service__workflow__url | <http||https>://\<service-host>/api/workflow/v1 |
Optional Airflow Variables
| Variable name | Example | Description |
|---|---|---|
| core__ingestion__batch_save_enabled | true/false | Enable batch saving in Storage |
| core__ingestion__batch_save_size | 400 | Size of batches to save in Storage (default: 500) |
| core__ingestion__thread_save_number | 10 | Number of simultaneous writings in Storage |
Backward Compatibility
Airflow 1.10.15 is as a “bridge” release but in OSDU Airflow 1.10.10 version should be supported.
Installation from source
-
Pull the latest changes from https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib
-
Use Python 3.6. Also, it is highly recommended using an isolated virtual environment for development purposes (Creation of virtual environments: https://docs.python.org/3.6/library/venv.html)
-
Make sure you have setuptools and wheel installed
pip install --upgrade setuptools wheel- Change directory to the root of the project
cd path/to/osdu-airflow-lib- Make sure osdu-airflow isn't already installed
pip uninstall osdu-airflow- Install OSDU Airflow
python setup.py installExample import after installing:
from osdu_airflow.backward_compatibility.default_args import update_default_argsInstallation from Package Registry
pip install 'osdu-airflow' --extra-index-url=https://community.opengroup.org/api/v4/projects/668/packages/pypi/simplePackage Lifecycle
The project can be deleted once Airflow 1.10.10 version support will be deprecated and no any additional logic will be added.
Licence
Copyright © Google LLC Copyright © EPAM Systems
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. A package to interface with OSDU microservices