M24 Eds Scheduler DAG doesn't support multi-partition

This issue has been identified during the validation of the M24 eds_scheduler DAG multi-tenancy functionality. Specifically, when the trigger_eds_ingest task in the eds_scheduler DAG dispatches the eds_ingest DAG, the fetch_client task is skipped for one partition while it executes successfully for another.

Reproduction Steps:

Assume the environment has two partitions: partition1 and partition2.

  1. Set up the fetch and ingest workflow in partition1 (e.g., with CSDJ).
  2. The eds_scheduler dispatches the eds_ingest DAG for partition1, and the fetch_client task executes successfully.
  3. Set up the fetch and ingest workflow in partition2 (e.g., with CSDJ).
  4. The eds_scheduler dispatches the eds_ingest DAG for both partition1 and partition2. The fetch_client task executes successfully for partition1 but is skipped for partition2.
  5. Delete the CSDJ in partition1.
  6. The eds_scheduler dispatches the eds_ingest DAG for partition2, and the fetch_client task executes successfully for partition2.

Note: Between step 4 and step 5, if the eds_ingest DAG is manually triggered with the same CSDJ, the fetch_client task executes successfully.

Assignee Loading
Time tracking Loading