Skip to content
Snippets Groups Projects

Update VDS Specification

Merged Morten Ofstad requested to merge feature/morten.ofstad/DocumentationUpdate into master
4 files
+ 48
21
Compare changes
  • Side-by-side
  • Inline
Files
4
@@ -7,6 +7,22 @@ Cloud Storage Format
When VDS data are stored in a cloud object store, such as AWS S3 or Azure BLOB Store, each chunk is represented by a
separate object. A VDS will therefore consist of numerous separate objects, rather than a single file.
Chunk-metadata Pages
--------------------
In order to allow efficient detection of constant value chunks without the overhead of making a read/request, the chunk-metadata
can also be stored as a collection of pages with a fixed number of entries per page (defined in the LayerStatus JSON object) and
each of these pages is written as a separate object. For Wavelet compressed layers, the size of the chunk object is prepended to the chunk-metadata when storing the entry in the chunk-metadata page. This is used
when interpreting the adaptive levels metadata in order to figure out how much of the chunk data is necessary to download in order to decompress a particular level.
In order to support sparse VDSs, there is an optional page directory in the layer status
which gives the stored index of each page (which can be different from the page's index) or -1 if no page exists (which means no chunks on that page have been written). If the layer status indicates that
the layer has chunk-metadata pages but no page directory exists, all metadata pages are assumed to exist as object with their index as the name of the object.
Layers that don't have chunk-metadata pages are required to store chunk-metadata in each chunk object using the metadata mechanism of the cloud object store. It is an error
to have mismatching chunk-metadata in the chunk object and the chunk-metadata page, usually caused by an incomplete write/copy of the VDS. In order to fix this error, the chunk-metadata page
will have to be updated with the chunk-metadata from the chunk object.
URL Encoding
------------
@@ -14,4 +30,4 @@ All VDS objects will be accessed from a base URL, usually formed from a bucket U
URL you can get the VolumeDataLayout and LayerStatus JSON objects by appending ``/VolumeDataLayout`` and
``/LayerStatus``. The binary chunk data are available through appending the layer name (found in the LayerStatus object,
e.g. Dimensions_012LOD0) and chunk index to the base URL. The binary chunk metadata is available through the layer name
with ``/ChunkMetadata`` and the metadata page index appended.
\ No newline at end of file
with ``/ChunkMetadata`` and the stored metadata page index appended.
Loading