Unexpected errors when accessing dataset with OpenVDS 3.4.0
Hi,
I am currently testing out OpenVDS 3.4.0 and I really appreciate the increase amount of error messages. (Un)fortunately, I am also hit by some of the error messages. I am not sure if I am doing anything wrong or if our data set in the Azure storage account is broken. The error messages I see do not appear when using in OpenVDS 3.3.3.
I have attached some Python scripts that I used to create the error messages at the bottom of this issues. I can also split this issue into several if it is too much content.
Any hints to what is going wrong and on how to debug this would be greatly appreciated.
Versions, storage, dataset
-
OpenVDS 3.4.0
-
Azure blob store
-
Dataset: Volve
ST10010ZC11_PZ_PSDM_KIRCH_FULL_T.MIG_FIN.POST_STACK.3D.JS-017536
with RLE compression and brick size 32. The file was created in June 2023 with OpenVDS/SEGYImport, but cannot tell which OpenVDS version it was.This is the only file we did some testing on. I noticed that there are a few blobs with size 0. I am not sure if this is relevant.
Observed errors
Invalid headers
I see several messages about invalid headers, but I am not sure which parts is invalid. I don't understand why this only affects a few blobs (2-4) and not all blobs.
[...]
Request was canceled.
OpenVDS reported error:
Invalid header (e.g. unsupported Wavelet compression version) for chunk: Dimensions_012LOD0/2647
[...]
Invalid HTTP request range (HTTP 416 error) and logging of SAS token
- The range header of the HTTP request seems to be wrong.
- The error message exposes the SAS token. I am not sure if this is done on purpose. At least I see the risk that the token may appear unexpectedly in log files
[...]
Http error response: 416 -> https://STORAGEACCOUNT/volve/ST10010ZC11_PZ_PSDM_KIRCH_FULL_T.MIG_FIN.POST_STACK.3D.JS-017536/vds_32_RLE/Dimensions_012LOD0/6945?SASTOKEN: The range specified is invalid for the current size of the resource.
RequestId: REQUESTID
Time:2024-03-06T13:27:02.0923545Z
[...]
I retracted potentially sensitive information.
None of the blobs mentioned in the error message has zero size:
Blob name, Access tier, Blob type, Size
7747, Hot (Inferred), Block blob, 68.04 KiB
5470, Hot (Inferred), Block blob, 128.04 KiB
4302, Hot (Inferred), Block blob, 128.04 KiB
I could reproduce this error if requesting VolumeSamples so far. My script for request VolumeSubsets hangs after inline 42 and several error messages about invalid headers.
Python script for reporduction
Below you find the Python code to request data from the VDS data set. The Python script needs the Python packages openvds
, python-dotenv
and numpy
.
Requesting subsamples
#!/usr/bin/env python3
import os
import numpy as np
import openvds
from dotenv import load_dotenv
load_dotenv()
SAS_TOKEN = os.getenv("SAS_TOKEN")
BLOB_URL = os.getenv("BLOB_URL")
if __name__ == "__main__":
print("Accessing samples from Azure blob")
try:
with openvds.open(url=BLOB_URL, connectionString=SAS_TOKEN) as vds:
manager = openvds.getAccessManager(vds)
layout = manager.volumeDataLayout
# Voxel extents are hardcoded for volve cube
# ST10010ZC11_PZ_PSDM_KIRCH_FULL_T.MIG_FIN.POST_STACK.3D.JS-017536
for depth in range(850):
samplePositions = []
for il in range(401):
for xl in range(720):
samplePositions.append((depth + 0.5, xl + 0.5, il + 0.5))
samplePositions = np.array(samplePositions)
print(f"Request at depth voxel {depth+0.5}")
request = manager.requestVolumeSamples(
samplePositions=samplePositions,
dimensionsND=openvds.DimensionsND.Dimensions_012,
lod=0,
channel=0,
interpolationMethod=openvds.InterpolationMethod.Cubic,
)
success = request.waitForCompletion()
if success:
print("Request was successfull")
else:
print(f"request:\n{request}")
if request.isCanceled:
print("Request was canceled.\n OpenVDS reported error:\n"
f" {request.errorMessage}"
)
else:
print("Request failed due to timeout")
except RuntimeError as error:
print(f"Could not open VDS: {error}")
Requesting subsets
#!/usr/bin/env python3
import os
import numpy as np
import openvds
from dotenv import load_dotenv
load_dotenv()
SAS_TOKEN = os.getenv("SAS_TOKEN")
BLOB_URL = os.getenv("BLOB_URL")
if __name__ == "__main__":
print("Accessing samples from Azure blob")
try:
with openvds.open(url=BLOB_URL, connectionString=SAS_TOKEN) as vds:
manager = openvds.getAccessManager(vds)
layout = manager.volumeDataLayout
sampleDimension, crosslineDimension, inlineDimension = (0, 1, 2)
for inlineIndex in range(401):
voxelMin = (0, 0, inlineIndex)
voxelMax = (
layout.getDimensionNumSamples(sampleDimension),
layout.getDimensionNumSamples(crosslineDimension),
inlineIndex + 1,
)
buffer = np.empty(
(
layout.getDimensionNumSamples(crosslineDimension),
layout.getDimensionNumSamples(sampleDimension),
)
)
print(f"Request at inline {inlineIndex}")
request = manager.requestVolumeSubset(
data_out=buffer,
dimensionsND=openvds.DimensionsND.Dimensions_012,
min=voxelMin,
max=voxelMax,
lod=0,
channel=0,
)
success = request.waitForCompletion()
if success:
print("Request was successfull")
else:
print(f"request:\n{request}")
if request.isCanceled:
print("Request was canceled.\n OpenVDS reported error:\n"
f"{request.errorMessage}"
)
else:
print("Request failed due to timeout")
except RuntimeError as error:
print(f"Could not open VDS: {error}")