M24 Eds Scheduler DAG doesn't support multi-partition
This issue has been identified during the validation of the M24 eds_scheduler DAG multi-tenancy functionality. Specifically, when the trigger_eds_ingest task in the eds_scheduler DAG dispatches the eds_ingest DAG, the fetch_client task is skipped for one partition while it executes successfully for another.
Reproduction Steps:
Assume the environment has two partitions: partition1 and partition2.
- Set up the fetch and ingest workflow in partition1 (e.g., with CSDJ).
- The
eds_schedulerdispatches theeds_ingestDAG for partition1, and thefetch_clienttask executes successfully. - Set up the fetch and ingest workflow in partition2 (e.g., with CSDJ).
- The
eds_schedulerdispatches theeds_ingestDAG for both partition1 and partition2. Thefetch_clienttask executes successfully for partition1 but is skipped for partition2. - Delete the CSDJ in partition1.
- The
eds_schedulerdispatches theeds_ingestDAG for partition2, and thefetch_clienttask executes successfully for partition2.
Note: Between step 4 and step 5, if the eds_ingest DAG is manually triggered with the same CSDJ, the fetch_client task executes successfully.