Review Task: ADR - DAG Migration Plan to CIMPL

Review Task: ADR - DAG Migration Plan to CIMPL

Issue: ingestion/home #56 — Draft: [ADR] DAG migration plan to CIMPL

Objective

Review and improve ingestion/home #56 so the ADR is consistent, decision-complete, and ready to advance from Initiated to Proposed / Under review. Update the issue body, status/labels, and tracking sections; leave concrete open questions as comments where a design choice still needs consensus.

Scope — issue to review

Field Value
Issue ingestion/home #56
Repo osdu/platform/data-flow/ingestion/home
Title Draft: [ADR] DAG migration plan to CIMPL
Author Yan Sushchynski [EPAM/GC]
Assignees none
Labels ADR::Proposed, M28, Venus
Milestone M28
Confidential Yes
Status checklist (current) Only Initiated checked

Context

ingestion/home #56 is a draft Lightweight ADR proposing migration of OSDU ingestion DAGs (and shared libraries) to CImpl.

In-scope DAGs:

Shared libraries:

Decision (as drafted):

  1. Make common-python-sdk CImpl-compatible or transition to osdu-python-client
  2. Remove template/render scripts; one cloud-agnostic DAG definition
  3. Design a cloud-agnostic pattern for Kubernetes Pod Operator config injection (SEGY converters)

Phase 1 pilots (named in ADR): ingestion-dags and segy-to-zgy-conversion.

What to verify / improve

  1. Status consistency — Title says Draft; label is ADR::Proposed; Status checklist only has Initiated. Align title, label, and checkboxes before soliciting review.
  2. Decision ambiguity (SDK path) — Decision item 1 is an either/or. Pick a preferred path and define exit criteria for switching to the alternative.
  3. Ownership and sequencing — Add pilot order, owners, and dependency order (SDK/libs before DAG consumers).
  4. K8s Pod Operator pattern — Underspecified. List concrete options (e.g. env-from ConfigMap/Secret, runtime CSP lookup, shared operator wrapper) and decision criteria.
  5. Timeline TBDs — Replace “TBD” with target dates or M28-linked milestones for Proposed and Phase 1.
  6. Related work — Add a “Related issues/MRs” section; cross-link known Venus/M28 / AF3 / CImpl DAG work already in flight.
  7. Consequences gaps — Call out consumer impact (CSP render scripts, existing CSP forks) and rollback / compatibility approach.
  8. Confidentiality — Confirm whether the ADR should remain confidential once Proposed; open for community review if appropriate.
  9. Assignees / reviewers — Issue has no assignees. Assign author plus CSP/ingestion maintainers for review.

Suggested improvements

Apply these edits directly on ingestion/home #56:

1. Align status metadata

Current Suggested
Title: Draft: [ADR] DAG migration plan to CIMPL [ADR] DAG migration plan to CIMPL (drop Draft once Proposed)
Label: ADR::Proposed Keep only if Status checklist moves to Proposed
Status: Initiated only Check Proposed (and Under review when reviewers are assigned)

2. Resolve Decision item 1

Replace the open “or” with something like:

(Or invert preferred/fallback — but choose one.)

3. Add sequencing section

Suggested outline:

  1. Shared libs / SDK CImpl readiness (common-python-sdk, osdu-ingestion-lib, osdu-airflow-lib)
  2. Pilot A: ingestion-dags (no Pod Operator complexity)
  3. Pilot B: segy-to-zgy-conversion (Pod Operator pattern)
  4. Revisit ADR → remaining DAGs

Name owners per pilot and per library.

4. Expand Pod Operator decision

Add alternatives and criteria, for example:

Option Pros Cons
Env-from ConfigMap/Secret at deploy time Simple; CSP injects values outside DAG Still CSP-specific deploy artifacts
Runtime CSP lookup via shared helper DAG code stays agnostic Needs shared contract + tests
Shared Pod Operator wrapper in osdu-airflow-lib One place to maintain Library release coupling

Require a chosen option (or a spike issue) before Phase 1 pilot B starts.

5. Replace timeline TBDs

Gate Current Suggested
Proposed & Under Review TBD — Next 1–2 weeks Concrete date or “by end of M28 sprint N”
Phase 1 (Pilot Migration) TBD — Next 1–2 months Target milestone / date + success criteria

6. Add Related issues/MRs section 

## Related issues / MRs

- (link AF3 / CImpl / Venus work as known)
- Phase 1 pilot issues: TBD
- SDK / library tracking issues: TBD

7. Strengthen Consequences

Add under Negative / operational:

  • CSP consumers of render scripts must migrate to the single DAG definition
  • Temporary dual-path support (render vs CImpl) — yes/no, and for how long
  • Rollback plan if Phase 1 pilots fail CImpl validation

8. Confidentiality and ownership

  • Decide: keep confidential vs make public for community ADR review
  • Assign: author + at least one ingestion maintainer and CSP stakeholders

Deliverables from reviewer

  1. Updated issue description on ingestion/home #56 covering status alignment, preferred SDK path, sequencing, Pod Operator options, timelines, related links, and consequences
  2. Status/label/title consistency — Initiated → Proposed (and Under review when ready)
  3. Assignees / reviewers set on the issue
  4. Comment listing any remaining open questions that need consensus before Approval
  5. Confidentiality decision documented in the issue (keep confidential or open)
Edited by Pavel Kiadrynski (EPAM)