Eds pipeline scanner

Background:

As part of OSDU InfoSec requirements, the pipeline should have Security and Secret Scanner built within the CICD. The pipeline is required to pass to ensure ease of monitoring and ensuring good quality of code are checked in

Code change

• Imports optimization and formatting
• Code formatting and best practices - return value type validation, python code formatting
• Upgrade deprecated library imports for Airflow

Dependent Merge Request:

Updates to the latest Airflow and Python version (compatible with M25) is required to ensure that we are using the latest image and packages for our pipeline

• Upgrade python-linters to support python 3.11 (!1225) · Merge requests · OSDU / OSDU Data Platform / CI-CD Pipelines · GitLab
• Upgrade airflow to 2.10.5 (!4) · Merge requests · OSDU / OSDU Data Platform / Deployment and Operations / PyType Container · GitLab

Related Issue:

• EDS Naturalization: Transition to Incubator state (#139) · Issue · osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow
• osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow#139 (comment 376213)

Results:

• All pipelines' tasks have passed if the dependent MR are merged in with the new version of Python 3.11 and Airflow 2.10.5