Use kubernetespodoperator

Type of change

  • Bug Fix
  • Feature

Does this introduce a change in the core logic?

  • [Yes]

Does this introduce a change in the cloud provider implementation, if so which cloud?

  • AWS
  • Azure
  • GCP
  • IBM

Updates description?

Currently the conversion DAG uses the KubernetesPodSegyToOpenVDSOperator operator from the osdu-airflow library to run the OpenVDS container with SEGYimport for conversion. The operator prepares arguments for SEGYimport and performs metadata updates. If there is a need to change what parameters are passed to SEGYimport, this library will have to be modified and redeployed. The same goes for changing what is passed to the DAG in the execution_context.

The changes in the merge request move the logic from the KubernetesPodSegyToOpenVDSOperator class to basic Python tasks within the DAG. It also extends the dagrun_timeout to avoid the DAG being aborted when converting large datasets.

Fixes #24

Edited by Asbjørn L. Johansen

Merge request reports

Loading