Skip to content

Increase chunk size for cloud read.

Paal Kvamme requested to merge kvamme62/consolidate-algo into master

Add code to increase performance by reading larger chunks at a time from the cloud.

When the application requests a region that covers more then one brick, the naive approach is to loop and read one brick at a time. The logic added in this merge request will consolidate reads that happen to be adjacent or nearly adjacent.

This will often be the case. Many applications will want to read full traces. And when creating a ZGY file it is recommended to order the bricks with the vertical direction varying fastest.

If the intended use of a ZGY file is to not read traces then the application producing the file is free to use a different layout. The performance benefit is seen whenever the read access pattern is similar to what was used on write.

The logic should also prove useful in the planned caching module.

The changes are not relevant for reading local files. In that scenario the buffer cache built into the operating system provides the same benefit for free.

Merge request reports