Newer
Older
OSDU Airflow Library is the package providing Airflow specific logic.
## Contents
- [OSDU Airflow Library](#osdu-airflow-library)
- [Contents](#contents)
- [Getting Started](#getting-started)
- [Required Airflow Variables](#required-airflow-variables)
- [Optional Airflow Variables](#optional-airflow-variables)
- [Backward Compatibility](#backward-compatibility)
- [Installation from source](#installation-from-source)
- [Installation from Package Registry](#installation-from-package-registry)
- [Package Lifecycle](#package-lifecycle)
- [Licence](#licence)
## Required Airflow Variables
The following variables are used by `osdu_api` clients. If some of them are missing `osdu_api` will try to find corresponding values in `osdu_api.ini` file, which path is set in `OSDU_API_CONFIG_INI` environmental variable.
|Variable name|Example|
|-------------|-------|
|core__service__dataset__url | \<http\|\|https\>://\<service-host\>/api/dataset/v1 |
|core__service__file__host | \<http\|\|https\>://\<service-host\>/api/file |
|core__service__file__url | \<http\|\|https\>://\<service-host\>/api/file/v2 |
|core__service__partition__url | \<http\|\|https\>://\<service-host\>/api/partition/v1/ |
|core__service__schema__host | \<service-host\> |
|core__service__schema__url | \<http\|\|https\>://\<service-host\>/api/schema-service/v1 |
|core__service__search__host | \<service-host\> |
|core__service__search__url | \<http\|\|https\>://\<service-host\>/api/search/v2 |
|core__service__seismic__url | \<http\|\|https\>://\<service-host\>/api/seismic-store/v3 |
|core__service__storage__host | \<service-host\> |
|core__service__storage__url | \<http\|\|https\>://\<service-host\>/api/storage/v2 |
|core__service__unit__url | \<http\|\|https\>://\<service-host\>/api/unit/v2 |
|core__service__workflow__host | \<http\|\|https\>://\<service-host\>/api/workflow |
|core__service__workflow__url | \<http\|\|https\>://\<service-host\>/api/workflow/v1 |
## Optional Airflow Variables
|Variable name|Example|Description|
|-------------|-------|-----------|
|core__ingestion__raise_on_any_error | false | Flag if a pipeline should stop executing if any error occurs |
|core__ingestion__batch_save_enabled | true/false | Enable batch saving in Storage |
|core__ingestion__batch_save_size | 400 | Size of batches to save in Storage (default: 500) |
|core__ingestion__thread_save_number | 10 | Number of simultaneous writings in Storage |
## Backward Compatibility
Airflow 1.10.15 is as a “bridge” release but in OSDU Airflow 1.10.10 version should be supported.
## Installation from source
1. Pull the latest changes from https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib
2. Use Python 3.11. Also, it is highly recommended using an isolated virtual environment for development purposes
(Creation of virtual environments: https://docs.python.org/3.11/library/venv.html)
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
3. Make sure you have setuptools and wheel installed
```sh
pip install --upgrade setuptools wheel
```
4. Change directory to the root of the project
```sh
cd path/to/osdu-airflow-lib
```
5. Make sure osdu-airflow isn't already installed
```sh
pip uninstall osdu-airflow
````
6. Install OSDU Airflow
```sh
python setup.py install
```
Example import after installing:
```python
from osdu_airflow.backward_compatibility.default_args import update_default_args
```
## Installation from Package Registry
```sh
pip install 'osdu-airflow' --extra-index-url=https://community.opengroup.org/api/v4/projects/668/packages/pypi/simple
```
# Package Lifecycle
The project can be deleted once Airflow 1.10.10 version support will be deprecated and no any additional logic will be added.
## Licence
Copyright © Google LLC
Copyright © EPAM Systems
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
A package to interface with OSDU microservices