Skip to content
GitLab
Commit a57c7d3a authored by Paal Kvamme's avatar Paal Kvamme
Browse files

Merge branch 'kvamme62/update_support' into 'master' 

Incremental rebuild of low resolution data

See merge request !72
parents e4c2cc19 082b183e
Pipeline #43325 passed with stages
in 8 minutes and 18 seconds
Expand all Hide whitespace changes
Inline Side-by-side
README.md
View file @ a57c7d3a
......@@ -234,8 +234,8 @@ tests to write to.
|Package |linux|windows|read|write|update|seisstore|zfp compress|old compress|
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|OpenZGY/C++ |y|y|y|y|-|y|y|N/A|
|OpenZGY/C++ Python wrapper |y|-|y|y|-|y|y|N/A|
|OpenZGY/C++ |y|y|y|y|y|y|y|N/A|
|OpenZGY/C++ Python wrapper |y|-|y|y|y|y|y|N/A|
|OpenZGY/Python |y|y|y|y|N/A?|linux|y|N/A|
|ZGY-Public, ZGY-Cloud |y|y|y|y|y|y|N/A|y|
|Old Python wrapper |y|y|y|y|y|y|N/A|y|
......
doc/lowres.html
View file @ a57c7d3a
......@@ -396,254 +396,211 @@ Other issues: More work for apps.
C++.
</p>
<h2>Support for ZGY file partial update</h2>
<p>
We need to understand how OpenZGY clients are planning to use
the library before deciding how flexible we need to be when it
comes to updating of existing files. If supported at all.
If supported then the next question is how good we need to
be on incremental updating of low resolution bricks.
</p>
<p>
Applications wanting update support for a file stored on the
cloud should normally be able to delete and re-create the file
instead. Unless the file is being used as some kind of working
storage that is updated frequently. If the file is updated
<i>really</i> frequently it might even be needed to defer
updating low resolution data until actually accessed.
</p>
<p>
Switching to Cloud enabled ZGY access <b>cannot be made
completely transparent</b> to application code. Especially not
if updating parts of a file. The main problem is that neither
GCS buckets nor Azure storage allow updates. We can allow
updates in the OpenZGY API but each time the ZGY file is
updated the file size on the cloud will grow.
Another can of worms, outside the scope of this document,
is the need for <b>read and write locks</b> for files on the
cloud if ZGY files cannot be treated as immutable.
</p>
<p>
For similar reasons OpenZGY will also leak disk space if a
compressed file is allowed to be opened for update. For
compressed data this happens both in cloud storage and
on-prem.
</p>
<p>
Update support has no obvious issues for on-prem. But do bear
in mind one of the early design decisions: Avoid spending time
implementing something that will only work on-prem. Because
soon everybody will be 100% on the cloud anyway.
</p>
<p>
Supporting any kind of partial update means that OpenZGY will
either need to re-generate all the statistics, histogram, and
low resolution bricks, or it will need to do some kind of
incremental update. This is the topic of the explanation that
follows.
</p>
<table border="1" style="border-collapse: collapse">
<tr>
<td style="padding-top: 20px; padding-bottom: 20px">
<img src="update-lowres-fig2-1.png"
width="257" height="172" alt="Alternative 1"/>
</td>
<td>
<dl>
<dt style="padding-bottom: 20px; padding-left: 10px">
All low resolution bricks get updated.
All full resolution data needs to be read.
</dt>
<dd>
This is the existing solution. It assumes that the
application will not update the file. Or that updating
a file will usually change a large part of it. ISSUE:
Very slow if the applicaction wants to update a tiny
part of a very large file. Also leaks more disk space
than the alternative, although that can be fixed
relatively easy.
</dd>
</dl>
</td>
<td>
Tasks:
<ul>
<li>
Per brick, skip write if contents not changed.
</li>
</ul>
</td>
</tr>
<tr>
<td style="padding-top: 20px; padding-bottom: 20px">
<img src="update-lowres-fig2-2.png"
width="257" height="172" alt="Alternative 2"/>
</td>
<td>
<dl>
<dt style="padding: 20px">
Affected low resolution bricks get updated. Always
re-calculate whole low resolution bricks which means
that some unchanged data needs to be read. Usually
from different LOD levels.
</dt>
<dd>
Keep today's plan C which recursively calculates the
low resolution bricks top down. Short cut the algorithm
when all inputs are known to be unchanged. ISSUE:
the simplest implementation would do this per genlod
buffer which is larger than one brick.
ISSUE: Not usable for compressed files due to
accumulation of noise. Except possibly if LOD
levels 2 and above are stored uncompressed.
</dd>
</dl>
</td>
<td>
Tasks:
<ul>
<li>
Keep track of dirty bricks.
</li>
<li>
Implement the shortcut.
</li>
<li>
Shortcut per brick.
</li>
</ul>
</td>
</tr>
<tr>
<td style="padding-top: 20px; padding-bottom: 20px">
<img src="update-lowres-fig2-3.png"
width="257" height="172" alt="Alternative 3"/>
</td>
<td>
<dl>
<dt style="padding: 20px">
Only the affected low resolution samples get updated.
A read/modify/write is often needed when updating a
low resolution brick. Sometimes even twice. Old data
is still being read back but this time the read back is
from the bricks about to be updated. There will be less
read back in most cases; the figure is misleading
because it shows a 2d view. The main problem is the
bricks that need more than one read/modify/write.
</dt>
<dd>
Use a completely different algorithm (plan B) for
updates and might as well use this for all writes
to uncompressed files. Calculate bottom-up.
If we ever allow updating of compressed files
(unlikely) then this alternative will not work.
ISSUE: More code to write, maintain, and test.
ISSUE: Not usable for compressed files due to
accumulation of noise. Except possibly if LOD
levels 2 and above are stored uncompressed.
</dd>
</dl>
</td>
<td>
Tasks:
<ul>
<li>
Keep track of dirty bricks.
</li>
<li>
Implement plan&nbsp;B.
</li>
</ul>
</td>
</tr>
<tr>
<td style="padding-top: 20px; padding-bottom: 20px">
<img src="update-lowres-fig2-4.png"
width="257" height="171" alt="Alternative 4"/>
</td>
<td>
<dl>
<dt style="padding: 20px">
As alternative 3 but somehow detect which bricks
receive multiple updates, that for those somehow
choose a different algorithm that avoids this.
</dt>
<dd>
See alternatives 2 and 3.
</dd>
</dl>
</td>
<td>
Tasks:
<ul>
<li>
Keep track of dirty bricks.
</li>
<li>
Implement plan&nbsp;B.
</li>
<li>
Implement additional logic.
</li>
</ul>
</td>
</tr>
<tr>
<td style="padding-top: 20px; padding-bottom: 20px">
<img src="update-lowres-fig2-5.png"
width="215" height="152" alt="Alternative 5"/>
</td>
<td>
<dl>
<dt style="padding: 20px">
Introduce a new &quot;Virtual ZGY&quot; file format to
be used for very large files. This would basically be
a list of regular ZGY files to read the real data
from. OpenZGY would hide that detail from the API. The
individual real ZGY files would be assumed to be small
enough to be deleted and re-created if any part of them
is touched. Incidentally this feature might also
facilitate allowing multiple processes write to the
(locically) same ZGY file.
</dt>
<dd>
Minor note: If it is desirable to still provide all LOD
levels up to the one where only one brick is required
then the single topmost LOD in each partial file should
be stored uncompressed and should be duplicated in a
single &quot;low-resolution&quot; partial file
holding all LOD levels that are built from data in
multiple files.
</dd>
</dl>
</td>
<td>
Tasks:
<ul>
<li>
Too many steps to list here.
</li>
</ul>
</td>
</tr>
</table>
<p>
Additional issue for all alternatives except the first one.
The code must handle incremental update of statistics and
histogram by reading the area to be overwritten and
subtracting its contribution before adding the new data. There
are issues with numerical accuracy and what happens if the new
value range is larger than the old so the histogram has to be
resized.
</p>
<p>
Additional issue for alternative 1: Updating data on the cloud
actually ends up appending data, wasting the space taken by
the old contents. This is how the cloud buckets work and is
not feasible to work around. So updates need to be kept to a
minimum. The algorithm in the first alternative can end up
overwriting some low resolution bricks which the exact same
content that they already have. So, additional code is needed
to prevent that.
</p>
<p>
OpenZGY allows an already written file to be opened for update
and have data appended or (uncompressed only) updated. This
raises the question of when and how low resolution data is
updated.
</p>
<p>
Applications wanting update support for a file stored on the
cloud should consider very carefully whether they want to use
the partial update feature. See the cloud-specific caveats
discussed below. Often it works better to just delete and
re-create the file instead. Unless the file is being used as
some kind of working storage that is updated frequently. If the
file is updated <i>really</i> frequently it might also be needed
to defer updating low resolution data until actually accessed.
</p>
<p>
When writing to the cloud there are several caveats. Switching
to Cloud enabled ZGY access <b>cannot be made completely
transparent</b> to application code. Especially not if updating
parts of a file. The main problem is that neither GCS buckets
nor Azure storage allow updates. We can allow updates in the
OpenZGY API but each time the ZGY file is updated the file size
on the cloud will grow. Another can of worms, outside the scope
of this document, is the need for <b>read and write locks</b>
for files on the cloud if ZGY files cannot be treated as
immutable.
</p>
<p>
For similar reasons OpenZGY will also leak disk space if a
compressed file is allowed to be opened for update. For
compressed data this happens both in cloud storage and on-prem.
</p>
<h2>Support for incremental build of low resolution data</h2>
<p>
This only applies to files where at least the low resolution
bricks are stored uncompressed. Otherwise the operation would
keep accumulating compression noise.
</p>
<p>
If a file to be updated has a valid set of low resolution bricks
the OpenZGY library will start tracking in memory changes to
histogram and statistics and maintaining a list of which bricks
have been touched. This means that after updating a file it may
be possibly to only re-calculate low resolution data in areas
that have changed. Currently the default is still to run a
complete rebuild on each close. Applications can request the
incremental option in the finalize() call. The change tracking
is not foolproof. Numerical inaccuracy might affect the
histogram. The histogram range cannot grow and the statistical
range cannot shrink.
</p>
<p>
The incremental algorithm is reasonably efficient. Whenever a
brick-column is processed and the flags say it is unchanged,
instead of calculating the contents recursively it will just
read the decimated data from the layer above. The fact that the
granularity is one brick-column instead of just one brick means
that some redundant computation might still be done. Also,
reading decimated data from the layer above means reading just
1/4 of a brick column. If there are 2 or 3 dirty "sibling
brick-columns", i.e. data that map to the same brick at lod+1,
then the same data gets read more than once. Fixing those issues
is probably not worth the effort.
</p>
<h2>Allow reading low resolution data from a file still open for write</h2>
<h4>Ambition level 1</h4>
<p>
Add a "lod" argument to IZgyWriter::read().
</p>
<p>
This is a trivial change now that support has been added to the
lower level code. There is even a unit test ready to be enabled
if the change is done. But due to several caveats the feature
should not be enabled unless somebody needs it.
</p>
<p>
The main caveat is that the low resolution data will only be up
to date after a call to finalize(). This puts an additional
burden on the application and a risk of subtle errors if the
application forgets.
</p>
<p>
Another caveat: The first time finalize() is called, the value
range of the histogram gets locked down to the range of samples
seen at that point. It gets unlocked again on a full finalize.
If the application calls the first finalize too early then this
means more subtle behavior the application needs to be aware of.
</p>
<p>
Another caveat is that if the file is on the cloud then every
finalize will leak some disk space.
</p>
<h4>Ambition level 2</h4>
<p>
Make this transparent to the application by doing an incremental
finalize each and every time the application reads lowres data
from a file open for write. This is not as expensive as it
sounds because if there has not been any writes between two or
more reads then the incremental builds the finalize is a no-op.
And of course, being incremental it won't be that expensive. But
higher level lods might be rebuilt fairly often and the single
brick that is the highest lod will be rebuilt on any change.
</p>
<p>
nlods() should be changed to return the possible number of lods
even when the lowres data is not current. Because it magically
becomes current when needed.
</p>
<p>
Caveat: The note in ambition level 1 about calling finalize()
too often is worse here since the application is no longer in
control of when finalize is called. As a minimum the finalize on
file close probably needs a full rebuild even when it otherwise
seems safe to do an incremental one. The code should set a
special flag to this effect if the change from "need full" to
"incremental allowed" is caused by one of these implicit
rebuilds. Calling close() then does a full build. On the cloud,
an arbitrary amount of space can be leaked because there is no
control of how many finalize() calls there will be.
</p>
<h4>Ambition level 3</h4>
<p>
Only generate the data actually needed to resolve the user's
read request. After thinking more closely about this I believe
the cost of coding is way higher than the benefits.
</p>
<p>
The current code assumes that if brick X is clean, the
corresponding 1/8th of the parent is up to date. This will no
longer be true. So another bit "half-dirty" in the dirty bitmap
needs to note that brick X is clean but its parent is not so if
this is a recursive call then X must be treated as dirty. I
think I need to set this in all bricks of the requested LOD that
were dirty. Because the in-memory buffer of decimated data will
be discarded (or just not computed). Bricks in recursively
computed lower layers won't need this special handling. I hope.
</p>
<p>
The improved algorithm shown below is still not optimal because
if new lowres bricks are required then _calculate() will write
them to file and then the read request will read back that same
data instead of having _calculate() somehow keep the data in
memory. Implementing that appears to be ridiculously
complicated. Both because not all the bricks might have needed a
rebuild and because the rebuild returns both more (full brick
columns) and less (because just one brick column at a time) data
then what the application asked for. So copying the result out
to the user's buffer might get pretty complicated.
</p>
<pre>
Actual algorithm:
On read of LOD > 0: Check whether the exact area of the read request is dirty.
IF dirty:
IF no pre-existing histogram range: set special full-rebuild-flag = true.
Instanciate a GenLodC.
IF ambition level == 2:
Invoke _calculate() on top level
ELSE:
Make list of brick-columns at lod overlapping the read request.
FOR EACH brick column:
Invoke _calculate() for this column, passing need_result = false.
Somehow set the special half-dirty in the approptiate places.
END IF dirty
</pre>
<h4>Ambition level 4</h4>
<p>
This is orthogonal to the rest.
</p>
<p>
Re-introduce the dynamically widening histogram. This fixes the
caveat of incorrect histograms but will be expensive to code.
</p>
<h2>Virtual ZGY files</h2>
<img src="update-lowres-fig2-5.png"
width="215" height="152" alt="Alternative 5"/>
<p>
A different approach both to partial updates and incremental
builds is to introduce a new &quot;Virtual ZGY&quot; file format
to be used for very large files. This would basically be a list
of regular ZGY files to read the real data from. Plus histogram
and statistics for the whole file.
</p>
<p>
Incidentally this feature might also facilitate allowing
multiple processes to write to the (logically) same ZGY file in
parallel.
</p>
<p>
The OpenZGY API would hide these details in IZgyReader so
application code wouldn't even know that multiple files were
involved. Writing such files would require application changes.
</p>
<p>
individual real ZGY files would be assumed to be small enough to
be deleted and re-created if any part of them is touched. And to
do a full rebuild of low resolution data in this file only.
</p>
<p>
Minor note: If it is desirable to still provide all LOD levels
up to the one where only one brick is required then the single
topmost LOD in each partial file should be stored uncompressed
and should be duplicated in a single &quot;low-resolution&quot;
partial file holding all LOD levels that depend on data from
multiple files.
</p>
</body>
</html>
native/src/api.cpp
View file @ a57c7d3a
......@@ -354,61 +354,34 @@ public:
// consecutively and the brick lookup table comes last.
result._header_size = _meta->oh().bricklupoff() + _meta->oh().bricklupsize();
const std::vector<std::array<std::int64_t,3>>& lodsizes = _meta->ih().lodsizes();
const std::vector<std::int64_t>& alphaoffsets = _meta->ih().alphaoffsets();
const std::vector<std::int64_t>& brickoffsets = _meta->ih().brickoffsets();
// This had been much simpler if I made a getBrickFilePositionByIndex.
// And possibly moving part of this code inside class LookupTable.
// I only need to iterate over the contents of alup, blup, bend
// but here I need to iterate over brick positiom and let class
// LookupTable convert that back to a raw index.
std::array<std::int64_t,3> size = _meta->ih().size();
const std::array<std::int64_t,3> bs = _meta->ih().bricksize();
for (std::int64_t lod= 0; lod < _meta->ih().nlods(); ++lod) {
for (std::int64_t ii = 0; ii < size[0]; ii += bs[0]) {
for (std::int64_t jj = 0; jj < size[1]; jj += bs[1]) {
LookupTable::LutInfo info =
LookupTable::getAlphaFilePosition
(ii/bs[0], jj/bs[1], lod,
lodsizes,
alphaoffsets, alup, /*aend,*/
bytesperalpha);
//std::cout<< zgydump_format(ii/bs[0], jj/bs[1], 0, lod, info) << "\n";
switch (info.status) {
case BrickStatus::Missing: result._alpha_missing_count += 1; break;
case BrickStatus::Constant: result._alpha_constant_count += 1; break;
case BrickStatus::Normal:
result._alpha_normal_count += 1;
break;
case BrickStatus::Compressed:
result._alpha_compressed_count += 1;
result._alpha_compressed_size += info.size_in_file;
break;
}
for (std::int64_t kk = 0; kk < size[2]; kk += bs[2]) {
LookupTable::LutInfo info =
LookupTable::getBrickFilePosition
(ii/bs[0], jj/bs[1], kk/bs[2], lod,
lodsizes,
brickoffsets, blup, bend,
bytesperbrick);
//std::cout << zgydump_format(ii/bs[0], jj/bs[1], kk/bs[2], lod, info) << "\n";
switch (info.status) {
case BrickStatus::Missing: result._brick_missing_count += 1; break;
case BrickStatus::Constant: result._brick_constant_count += 1; break;
case BrickStatus::Normal:
result._brick_normal_count += 1; break;
case BrickStatus::Compressed:
result._brick_compressed_count += 1;
result._brick_compressed_size += info.size_in_file;
break;
}
}
}
for (std::size_t ix = 0; ix < alup.size(); ++ix) {
LookupTable::LutInfo info =
LookupTable::getAlphaFilePositionFromIndex(ix, alup, bytesperalpha);
switch (info.status) {
case BrickStatus::Missing: result._alpha_missing_count += 1; break;
case BrickStatus::Constant: result._alpha_constant_count += 1; break;
case BrickStatus::Normal:
result._alpha_normal_count += 1;
break;
case BrickStatus::Compressed:
result._alpha_compressed_count += 1;
result._alpha_compressed_size += info.size_in_file;
break;
}
}
for (std::size_t ix = 0; ix < blup.size(); ++ix) {
LookupTable::LutInfo info =
LookupTable::getBrickFilePositionFromIndex(ix, blup, bend, bytesperbrick);
switch (info.status) {
case BrickStatus::Missing: result._brick_missing_count += 1; break;
case BrickStatus::Constant: result._brick_constant_count += 1; break;
case BrickStatus::Normal:
result._brick_normal_count += 1; break;
case BrickStatus::Compressed:
result._brick_compressed_count += 1;
result._brick_compressed_size += info.size_in_file;
break;
}
size[0] = (size[0]+1)/2;
size[1] = (size[1]+1)/2;
size[2] = (size[2]+1)/2;
}
// TODO-Low: Keep track of wasted_size and padding_size.
......@@ -790,10 +763,12 @@ private:
compressor_t _lodcompressor;
public:
struct OpenForUpdate{};
/**
* \copydoc IZgyWriter::open()
*/
ZgyWriter(const ZgyWriterArgs& args)
explicit ZgyWriter(const ZgyWriterArgs& args)
: _fd()
, _accessor_rw()
, _dirty(false)
......@@ -846,7 +821,7 @@ public:
* See the ZgyReader constructor and the "truncate" ZgyWriter constructor.
* Understandably this nethod contains elements from both.
*/
ZgyWriter(const ZgyWriterArgs& args, bool update)
ZgyWriter(const ZgyWriterArgs& args, OpenForUpdate)
: _fd()
, _accessor_rw()
, _dirty(false)
......@@ -1193,8 +1168,6 @@ public:
/**
* \brief Remove the presumably stale statistics, histogram, and lowres.
* \details Currently ignores "action" because only full rebuild
* is supported.
*/
void _create_derived(
const std::vector<DecimationType>& decimation,
......@@ -1205,14 +1178,57 @@ public:
this->_dirty = false;
std::shared_ptr<InternalZGY::StatisticData> stats;
std::shared_ptr<InternalZGY::HistogramData> histo;
if (action == FinalizeAction::BuildIncremental) {
std::tie(stats, histo) = _accessor_rw->trackedChanges();
if (!stats || !histo) {
action = FinalizeAction::BuildFull;
//std::cout << "finalize: switched to full because no tracked changes."
// << std::endl;
}
else if (histo->getcount() <= 0) {
// Obscure case: The data has changed so much that none of the
// existing samples, not even defaultvalue, now fit inside the
// histogram. Be nice to applications (and unit tests) and
// just fall back to a full rebuild. See test_reopen() in
// test_reopen.cpp.
action = FinalizeAction::BuildFull;
//std::cout << "finalize: switched to full because of empty histogram."
// << std::endl;
}
else {
//std::cout << "finalize: will attempt incremental build." << std::endl;
}