Skip to content

Add support for updating existing files also on the cloud

Paal Kvamme requested to merge kvamme62/update-on-cloud into master

Cloud files can now be opened for update, with a few restrictions.

The segment size must be the same as when the file was created. The library can in some cases detect that the segment size is wrong and throw an exception. But it won't always be able to do so. Which is why the size isn't just set automatically.

It is recommended to finalize with FinalizeAction::Keep in all but the last update. For compressed files this is mandatory. The statistics, histogram, and low resolution data will then only be generated once.

If the application chooses to finalize more than once then performance can suffer and space might be wasted in the cloud file. The main mitigation for those problems is to use FinalizeAction::BuildIncremental instead of the default of FinalizeAction::BuildFull on every update. The reason incremental builds are not default is that statistics etc. might become slightly inaccurate. See "limitations" in the main README file. So this needs to be a conscious decision made by the application. Also the incremental build is not tested as thoroughly as the default.

Another mitigation for the residual wasted space is to use larger segment sizes. Just remember that the memory consumption will be segsize*segsplit. If the entire file fits in 5 or 6 segments there will be little or no wasted space. Large segment sizes (even if smaller than filesize/5) also help if the application writes misaligned data.

Merge request reports