Home issueshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues2023-07-05T10:09:40Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/48Parsers/converters code organization porposal2023-07-05T10:09:40ZSiarhei Khaletski (EPAM)Parsers/converters code organization porposal## Rationale
Now the number of different parsers was presented to the OSDU. The parsers/converters were implemented with using of different technologies and programming languages(C++, Java, Python, etc.).
It can cause difficulties duri...## Rationale
Now the number of different parsers was presented to the OSDU. The parsers/converters were implemented with using of different technologies and programming languages(C++, Java, Python, etc.).
It can cause difficulties during onboarding such parsers: requirements, code organization, runtime environment setup.
## Objective
Approve or develop a unified approach regarding to the parsers/converters representation and usage as Airflow DAGs’ operators.
## Proposal
The intention to use as much as possible containerized DAG’s steps, i.e. to use KubernetesPodOperator was mentioned as one of the best practices for Manifest-based Ingestion pipelines.
It means that the pipeline step can be implemented on absolutely different technology and executable part of the step will be executed inside Docker container.
The proposal is to deliver with a parser code a properly configured base Dockerfile. This docker file will contain only required dependencies to run the parser with ability to extend or configure the executable invocation (parameters, environment variables etc.)
Each CSP provider, if needed, should develop own Dockerfile with additional requirements or environment variables setup.
![Parsers_dependencies](/uploads/d24ee18309b2471c94bab6023422807d/Parsers_dependencies.png)
## Implementation
The proposal's implementation example - [WITSML parser](https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics-osdu-integration/-/tree/master/build)
### Note
For lightweight DAG’s dependencies (local dependencies) the [Packaged DAGs](https://community.opengroup.org/osdu/platform/data-flow/home/-/issues/47) approach can be used.Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/36Frame of Reference Normalization2021-02-25T12:44:48ZStephen Whitley (Invited Expert)Frame of Reference NormalizationTodd DixonTodd Dixonhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/8OpenVDS Enrichment2021-01-20T19:51:07ZStephen Whitley (Invited Expert)OpenVDS EnrichmentM1 - Release 0.1Marius Storm-Olsenmarius@bluware.comMarius Storm-Olsenmarius@bluware.com