Refactor DAG related code
Introduction
There is DAG related code that is executed in the container during a DAG run. The code is here and here. And this code looks messy and outdated, and requires some refactoring.
What should be done?
- Update the code to make it work with the most recent
osdu-*
Python libs. The dependencies are here https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/blob/master/build/requirements.txt - Delete deprecated functionality of processing files by
preload_file_path
here. - Add the static-analysis step in the CI/CD.
- Add possibility to pass the user's access/id token to the DAG
- Common refactoring, because the code is messy now (a lot of "ifs" and lines of code in a single function)