Airflow task are spending lot of time in scheduled state
Steps to reproduce: Upload 1000 small csv files (5 raw per CSV) and trigger SLB CSV workflow Expectation: Jobs should get completed in 10- 15 minutes. What actually happened: Jobs are taking hours to complete. Observation: Airflow task are waiting for long time (25-30 minutes) in scheduled state before getting queued. Once queued task takes few seconds to execute.
Please find our analysis here