Skip to content

Feature: read fast track

Yannick requested to merge feat/read_fast_track into master

upon some circumstances, perform read by taking some shortcuts without using Dask/ffspec but Pandas directly and async I/O from the blob storage. For now "fast track" read is disabled by default and could be enabled by setup env var READ_FAST_TRACK_ENABLE to true. Fast track read could perform up to 6x faster than regular read when dealing with bulk with many columns or small bulk. For more regular, gain is there but less important.

Merge request reports