Skip to content
Snippets Groups Projects
Normal view Permalink
README.md 696 B
Newer Older
  • Learn to ignore specific revisions
    • Dmitriy Rudko's avatar
    Publish manifest-based ingestion workflow
    Dmitriy Rudko committed
    1 2 3 4 5 6 7 8 
    The image johnybear/osdu-airflow:test with preinstalled Airflow and osdu-python-sdk is used in the pipeline. DAG tests are run inside this container.

To mock external APIs the Flask web-server is used. By default the address of the server is localhost:5000. Endpoints /storage and /workflow/updateStatus correspond to external endpoints.

The script test_dags.py triggers DAGs and checks periodically a triggered dag state until it stops having the running state; if valid data is passed script expects no failed state.
If we assume that a dag must fail the script will expect the fail state.

If it is unexpected result logs will be saved as job artifacts, so we can read them to find a problem