Open VDS issueshttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues2023-03-27T06:24:30Zhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/180openvds fails to read from s3 bucket2023-03-27T06:24:30ZGeorge Zavitsanosopenvds fails to read from s3 bucketI am using openvds-3.2.1 python package from PyPI. I am trying to connect and read a .vds format file located in a AWS S3 bucket, in which I have full access.
But the below code, fails:
uri = "s3://bucket-name/path-to-vds-file/filename...I am using openvds-3.2.1 python package from PyPI. I am trying to connect and read a .vds format file located in a AWS S3 bucket, in which I have full access.
But the below code, fails:
uri = "s3://bucket-name/path-to-vds-file/filename.vds"
openvds.open(uri)
with the error:
RuntimeError: Error on downloading VolumeDataLayout object: Http error respons: 404 -> https://bucket-name.s3.eu-central 1.amazonaws.com/path-to-vds-file/filename.vds/VolumeDataLayout
Reading vds files is working smoothly in my local environment.
Any help would be really appreciated. Thanks.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/178Installation on MacOS does not install libuv (OpenVDS 3.2.1)2023-04-03T08:34:43ZAlexander JaustInstallation on MacOS does not install libuv (OpenVDS 3.2.1)Thank you so much for updating the installation process! The actual compile process worked for me without any problems. :thumbsup:
## Description
I install OpenVDS in a non-standard installation prefix (I will call it `INSTALLATION_DI...Thank you so much for updating the installation process! The actual compile process worked for me without any problems. :thumbsup:
## Description
I install OpenVDS in a non-standard installation prefix (I will call it `INSTALLATION_DIR`). After the compilation and installation step I get the following error message when using tools like `VDSInfo` or `SEGYImport`:
```
$ VDSInfo --help
dyld[19054]: Library not loaded: @rpath/libuv.1.dylib
Referenced from: <B3C82796-5CB5-3C06-97AE-C111D8D19A9C> /Users/AEJ/software/openvds-3.2.1-install-python/lib/libopenvds.3.2.1.dylib
Reason: tried: '/Users/AEJ/software/openvds-3.2.1-install-python/lib/libuv.1.dylib' (no such file), '$ORIGIN/libuv.1.dylib' (no such file), '$ORIGIN/libuv.
1.dylib' (no such file), '$ORIGIN/../lib64/libuv.1.dylib' (no such file), '$ORIGIN/../lib64/libuv.1.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS@rpath/libuv.1.dylib' (no such file), '$ORIGIN/libuv.1.dylib' (no such file), '$ORIGIN/libuv.1.dylib' (no such file), '$ORIGIN/../lib64/libuv.1.dylib' (no
such file), '$ORIGIN/../lib64/libuv.1.dylib' (no such file), '/usr/local/lib/libuv.1.dylib' (no such file), '/usr/lib/libuv.1.dylib' (no such file, not in dyld cache)
Abort trap: 6
```
## Manual fix
I could fix the problem by:
1. Copy `libuv.1.0.0.dylib` from `${BUILD_DIR}/libuv_1.44.2_install/Release/lib/libuv.1.0.0.dylib` to `${INSTALLATION_DIR}/lib`.
2. Create a symlink within the ``${INSTALLATION_DIR}/lib/` directory such that `libuv.1.dylib -> libuv.1.0.0.dylib`.
Now the error is gone
```
$ VDSInfo --version
VDSInfo - OpenVDS 3.2.1 - Revision: 6922d9151901e3d727b31c776559dd033add30fd
```
## Compilation command
```
cmake -S . \
-B build \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_SHARED_LIBS=ON \
-DBUILD_JAVA=OFF \
-DBUILD_PYTHON=ON \
-DBUILD_EXAMPLES=ON \
-DBUILD_TESTS=OFF \
-DBUILD_DOCS=OFF \
-DDISABLE_AWS_IOMANAGER=ON \
-DDISABLE_AZURESDKFORCPP_IOMANAGER=ON \
-DDISABLE_GCP_IOMANAGER=ON \
-DDISABLE_DMS_IOMANAGER=ON \
-DDISABLE_STRICT_WARNINGS=ON \
-DCMAKE_FIND_FRAMEWORK=LAST \
-DAUTO_ADJUST_UUID=OFF \
-DBUILD_CURL=OFF \
-DCMAKE_MACOSX_RPATH=ON \
-DCMAKE_INSTALL_PREFIX="${INSTALLATION_DIR}"
```
## System
* Arm64 MacOS 13.2.1
* OpenVDS 3.2.1
* clang 14.0.0
```
$ clang --version
Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: arm64-apple-darwin22.3.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
```
* cmake 3.26.0 (via Homebrew)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/177Memory leak when using openvds-python to access wavelet-compressed vds in linux2023-03-27T06:41:31ZXudong DuanMemory leak when using openvds-python to access wavelet-compressed vds in linuxwhen I use the openvds-python lib to open a vds, access data, and close it, there is a problem of memory leak.
In linux, I open and close a wavelet-compressed vds severial times in one python script, the memory usage will increase rapid...when I use the openvds-python lib to open a vds, access data, and close it, there is a problem of memory leak.
In linux, I open and close a wavelet-compressed vds severial times in one python script, the memory usage will increase rapidly until the python process ends.
In windows, there is no problem.
In linux, there is no problem when open a uncompresssed vds.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/176VDS with LOD levels unexpectedly large2023-03-22T14:43:59ZAlexander JaustVDS with LOD levels unexpectedly largeI find VDS data sets to increase much more in size when LOD levels are added than expected.
## Obervation/Experiment
I created a VDS data sets from a volve data set (`ST0202R08_PS_PSDM_FULL_OFFSET_PP_TIME.MIG_FIN.POST_STACK.3D.JS-01753...I find VDS data sets to increase much more in size when LOD levels are added than expected.
## Obervation/Experiment
I created a VDS data sets from a volve data set (`ST0202R08_PS_PSDM_FULL_OFFSET_PP_TIME.MIG_FIN.POST_STACK.3D.JS-017534.segy`, about 1GiB) for different LOD levels. This is a 3D post-stack dataset such that I would expect an increase in file size of about 15% if all LOD levels are created. The major increase should appear for the first few LOD levels.
I observe the following file sizes:
| LOD level | Size in MiB | Relative size |
|-----------|-------------|---------------|
| 0 | 864,04 | 100% |
| 1 | 1248,06 | 144% |
| 2 | 1456,09 | 169% |
| 3 | 1456,09 | 169% |
| 4 | 1454,31 | 168% |
| 5 | 1456,09 | 169% |
| 6 | 1456,09 | 169% |
| 7 | 1456,09 | 169% |
| 8 | 1456,09 | 169% |
| 9 | 1454,88 | 168% |
| 10 | 1456,10 | 169% |
| 11 | 1456,10 | 169% |
| 12 | 1456,11 | 169% |
Now I see that the file size increases by more than 50%. If I scale my estimate I see that the increase at least flattens out quickly what agrees with the geometric series. However, I also see a small dip for LOD level 10 and 11 which I don't exactly understand, but maye that is due to some collapsed blocks.
I used the default settings, but also tested with LOD level and compression set explicitly with the same results for the file size. The file is stored on a local hard drive and file size is checked with `du`.
I also tested with a larger SEGY where VDS with no LOD levels is 11 GiB large. When adding 4 LOD levels the file size goes up to 20 GiB. This is an increase of file size by even more than **80%**.
## Questions
Is this increase in file size expected? I assumed that the file size increase should be according to the [geometric series](https://en.wikipedia.org/wiki/Geometric_series) with `a=1` and `r=(1/2)^d` which would be `r=1/8` for 3D data sets.
## Platform
* Apple Arm M1 Max
* MacOS 13.2.1
* OpenVDS 3.0.3 (compiled from source)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/175Downsampling for LOD levels wrong for first point2023-03-07T10:19:02ZAlexander JaustDownsampling for LOD levels wrong for first pointI have been playing around with LOD level generation on a VDS with synthetic content. My goal is to understand the LOD levels and the downsampling better.
## Observation/Experiment
I wrote a Python script ([test_lod_levels_sine_functio...I have been playing around with LOD level generation on a VDS with synthetic content. My goal is to understand the LOD levels and the downsampling better.
## Observation/Experiment
I wrote a Python script ([test_lod_levels_sine_function.py](/uploads/f7a3eb217cdba359ea77b0b7c96ec0a4/test_lod_levels_sine_function.py)) which generates a sine function (with specified frequency and amplitude etc) that is written to a 3D VDS. I let OpenVDS add 4 LOD levels such that I have 5 levels in total. In the current setup in the script, I have a single brick with 32 samples in each direction on level 0.
In the next step I load the VDS file and extract the data along a line. The data is plotted it against the analytical function and I compute some error norms (only printed on screen). See the following plot:
## Observations and questions
From what I understand here seems to be the following:
- If data is sampled down, I kind only keep the second, fourth etc. sample on each level is kept. This looks somewhat like this
```text
level Samples
0 0 1 2 3 4 5 6 7 8 ...
1 0 2 4 6 8 ...
2 0 4 8 ...
```
Missing values indicate samples that are not available on the LOD level.
Is this understanding correct?
- At the moment, it looks to me that down sampling simply removes/ignores values that are in between the sample that are being kept. At least for to me it looks like this to me from the plot.
Is this understanding correct or do you do any kind of elaborate downsampling that includes some kind of anti-aliasing?
- From my experiments I expect that for my sine wave, I simply get bigger gaps between discrete points the higher go up per level. I plotted this for the all levels in my VDS:
![lod_level_sine_function](/uploads/7e94e7385e12e5b044569d64b1ecc2c0/lod_level_sine_function.png){width=60%}
My assumptions on how the downsampling is done, seem to hold mostly. However, for some reason the **first** sample on levels > 0 seems to be shifted (point at top left in the plot). The y-value here is close to 1 instead of 0.
If my interpretation of how the x-coordinate is determined, I would expect that all points also show the same offset. However, all other samples on a level of detail seem to match with a sample of level 0.
Am I doing something wrong here when determining the x-coordinate or does the downsampling have a bug here?
## Platform
* Apple Arm M1 Max
* MacOS 13.2.1
* OpenVDS 3.0.3 (compiled from source)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/174Addition of valid input for SEGYImport2023-02-23T13:23:58ZAlexander JaustAddition of valid input for SEGYImportI wonder what would be the correct the conversion of SEG-Y files to VDS using `SEGYImport` when the provided options by `SEGYImport` are not rich enough.
Example: I have an attribute map in a SEG-Y file. However, none of the allowed na...I wonder what would be the correct the conversion of SEG-Y files to VDS using `SEGYImport` when the provided options by `SEGYImport` are not rich enough.
Example: I have an attribute map in a SEG-Y file. However, none of the allowed names for the `Attribute` property (`--attribute-name`: Amplitude (default), Attribute, Depth, Probability, Time, Vavg, Vint, or Vrms) are fitting my needs. The attribute name `Attribute` would be too vague for my use case and the other allowed names do not fit either.
- Should I import the SEG-Y file to VDS and afterwards change the name? I am not sure if that is possible, cf. #173.
- Could I provide a patch that extends SEGYImport by the units, names etc. that I would need?
- Should I fork and create my own SEGYImport? I would really like to avoid this.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/173Updating information stored in a VDS (Are VDS datasets mutable?)2023-03-27T06:48:15ZAlexander JaustUpdating information stored in a VDS (Are VDS datasets mutable?)I was playing around with OpenVDS to figure out whether and to what extent VDS datasets are mutable or not. My big question is: What parts of a VDS dataset are mutable. If there are any parts that are mutable, what is the correct way to ...I was playing around with OpenVDS to figure out whether and to what extent VDS datasets are mutable or not. My big question is: What parts of a VDS dataset are mutable. If there are any parts that are mutable, what is the correct way to change these parts.
Either way it has different implications for certain workflows (to me). This concerns metadata as well as channel data stored within the VDS.
1. If a VDS dataset is always immutable, I can be sure that nobody with accidentally change/break a VDS dataset.
2. If a VDS dataset is mutable, I could update some fields if, e.g., `SEGYImport` does not accept certain names/units during ingestion and/or update data within the VDS, e.g. add fast slice, an additional channel or update data within a channel without recreating the VDS dataset from scratch.
I potentially do some "stupid" things here, but I also try to model a worst case scenario here like "what is the worst thing somebody can do wrong?".
I found an [older issue](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/28) which mentions the addition of LOD levels to a VDS, but it does not specify whether this would happen in-place or would create a new VDS dataset.
## Observations / Experiments
1. I am able to add additional channel data to an existing channel of a VDS dataset. The additional data seems to hide the initially available channel data.
For testing I created a small Python script: [create_and_change_vds_inplace_small.py](/uploads/72d056826ce7bf455f1b69ddf023fec2/create_and_change_vds_inplace_small.py). The script creates a small artifical VDS dataset with one channel. The script carries out the following steps.
- Create VDS dataset with all values in the channel are set to `1`.
- Close file hande such that it can be written to disk.
- Open the file and extract a slice, copy the slice data and close the file.
- Open the file and get an AccessManager, write the value `2*old_value` (`2` in this case) to the VDS dataset and close the file.
- Open the file and extract a slice, copy the slice data and close the file.
- Plot the slice data.
When I run the script that the data extracted from the VDS indeed changes. For the first slice I get constant `1` values and constant `2` values for the second time I extract a slice. I am not sure if one can still access the "old" data. The file size increases so it appears to me as if the old data is still stored in the VDS dataset.
Is this behavior intended? If so, how can I access the "old" data? Would it be possible to actually update the data without increasing the file size?
2. I tried to update the metadata. For that I wrote a [small C++ code](/uploads/013acdfdfe460bef0d5dbec4b517ee98/update_metadata.cpp) as C++ seemed to have (more?) direct access to the `MetadataWriteAccess` object than Python. The code basically opens a specified VDS dataset and replaces the `ImportTimeStamp` values with some unrealistic time stamp.
The code executes, but gives a segmentation fault (`Segmentation fault: 11`). From debugging I concluded that the segmentation fault rises when the VDS dataset is being closed. Calling the `SetMetadataString` function seems to be fine.
Is this behavior intended? I guess the segmentation fault should never happen, but to me it is not clear if that is an error when updating the VDS dataset or it is a side effect of illegally writing the metadata.
## Platform
* Apple Arm M1 Max
* MacOS 13.1
* OpenVDS 3.0.3 (compiled from source)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/172KnownChannelNames class2023-02-23T12:36:51ZMorten OfstadKnownChannelNames classThere should be a KnownChannelNames class with the names from the documentation (https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/vds/specification/Metadata.html#named-channels) -- For C++ this is foun...There should be a KnownChannelNames class with the names from the documentation (https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/vds/specification/Metadata.html#named-channels) -- For C++ this is found in GlobalMetadataCommon.h, but it is not available in Python/Java.Morten OfstadMorten Ofstadhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/171Angular interpolation does not return original values2023-06-15T16:21:27ZAlena ChaikouskayaAngular interpolation does not return original valuesHi,
I've been trying to understand openvds interpolation behavior and noticed the following.
Interpolation documentation says that
> The sampled value will be exactly equal to the original at the voxel center, regardless of the interpol...Hi,
I've been trying to understand openvds interpolation behavior and noticed the following.
Interpolation documentation says that
> The sampled value will be exactly equal to the original at the voxel center, regardless of the interpolation method used.
But I observe on my synthetic file [test.vds](/uploads/629b6b4d6052de8a6d2562395cff3721/test.vds) that this doesn't hold. For the voxel middle points 4 methods return same data (as I would expect), but angular interpolation returns something different.
```
#include <OpenVDS/OpenVDS.h>
#include <OpenVDS/VolumeDataLayout.h>
#include <OpenVDS/VolumeDataAccess.h>
#include <iostream>
#include <map>
int main(int argc, char *argv[]) {
/*
| xlines-ilines | 1 | 3 | 5 |
|---------------|--------------------|--------------------|--------------------|
| 10 | 100, 101, 102, 103 | 108, 109, 110, 111 | 116, 117, 118, 119 |
| 11 | 104, 105, 106, 107 | 112, 113, 114, 115 | 120, 121, 122, 123 |
*/
std::string url = "file://test.vds";
std::string connectionString = "";
OpenVDS::Error error;
OpenVDS::VDSHandle handle = OpenVDS::Open(url, connectionString, error);
OpenVDS::VolumeDataAccessManager accessManager = OpenVDS::GetAccessManager(handle);
OpenVDS::VolumeDataLayout const *layout = accessManager.GetVolumeDataLayout();
int sampleCount0 = layout->GetDimensionNumSamples(0);
int traceCount = 1;
// Angular is correct only for (1.5, 0.5)
float inlineValue = 2.5; // choose from 0.5, 1.5 and 2.5
float xlineValue = 1.5; // choose from 0.5 and 1.5
std::vector<float> buffer(traceCount * sampleCount0);
float tracePos[traceCount][6];
for (int trace = 0; trace < traceCount; trace++)
{
tracePos[trace][0] = 0;
tracePos[trace][1] = xlineValue;
tracePos[trace][2] = inlineValue;
tracePos[trace][3] = 0;
tracePos[trace][4] = 0;
tracePos[trace][5] = 0;
}
std::map<std::string, OpenVDS::InterpolationMethod> interpolations{
{"Nearest", OpenVDS::InterpolationMethod::Nearest},
{"Linear", OpenVDS::InterpolationMethod::Linear},
{"Cubic", OpenVDS::InterpolationMethod::Cubic},
{"Triangular", OpenVDS::InterpolationMethod::Triangular},
{"Angular", OpenVDS::InterpolationMethod::Angular}
};
for (const auto& interpolation : interpolations) {
auto request = accessManager.RequestVolumeTraces(
buffer.data(),
buffer.size() * sizeof(float),
OpenVDS::Dimensions_12, // I understand it doesn't matter if it is Dimensions_12 or Dimensions_012
0,
0,
tracePos,
traceCount,
interpolation.second,
0
);
request.get()->WaitForCompletion();
std::cout << "Interpolation " << interpolation.first <<": " ;
for (float f: buffer) {
std::cout << f << ' ';
}
std::cout << "\n" << std::flush;
}
}
```
(my openvds version is some 3.1-ish build, but I do not see anything about changes to angular interpolation in release notes, so I assume it's the same)
Am I doing something wrong?https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/170Allow VDSCopy to overwrite existing SDMS dataset: `--allow-overwrite`2023-03-15T08:22:49ZFilip BrzękAllow VDSCopy to overwrite existing SDMS dataset: `--allow-overwrite`Hello,
first of all, thank you very much for the great `3.1` release, and re-implementation of `IOManagers`, for SDMS, it's been very helpful as `sdapi` was troublesome.
While examining new IOManagers with `OPENVDS_DMS_CURL=1`, I have ...Hello,
first of all, thank you very much for the great `3.1` release, and re-implementation of `IOManagers`, for SDMS, it's been very helpful as `sdapi` was troublesome.
While examining new IOManagers with `OPENVDS_DMS_CURL=1`, I have noticed a change in the behavior, that prevents us from writing bulk trace data (VDS content) into the previously created dataset (with the metadata we require), that doesn't yet have any data loaded.
```shell
OPENVDS_DMS_CURL=1 AWS_REGION=us-east-1 VDSCopy ./data/syntethic_data.wavelet.vds sd://osdu/<test-project>/demo-1
[CURL http respons error 409. Automatic rety https://<REDACTED>/api/seismic-store/v3/dataset/tenant/osdu/subproject/<REDACTED>/dataset/demo-1?path=%2F]
...
[Could not create VDS sd://osdu/<REDACTED>/gsi-demo-1] Seismic dms lock failed: Http error respons: 409 -> https://<REDACTED>/api/seismic-store/v3/dataset/tenant/osdu/subproject/<REDACTED>/dataset/demo-1?path=%2F
- [seismic-store-service] The dataset sd://osdu/<REDACTED>/demo-1 already exists[seismic-store-service]
```
when it's run through old DMS flow (using `sdapi`), it's happy to ignore 409, and proceed to write it, command for the reference, but it's results in `seg-fault` at the end (it was reported here #123)
```
AWS_REGION=us-east-1 VDSCopy ./data/syntethic_data.wavelet.vds sd://osdu/<test-project>/demo-1
```
Is it possible to add `--allow-overwrite` similar to the flag available in `VDSUploader.sh` in HueSpace SDK, which would ignore 409, when the dataset was created previously?
Rationale: we want to have control over how the dataset is created in SDMS, for data-lineage reasons, for which we need to create it ourselves, and then populate sd location with VDS content.
Regards,
Filip
PS. Is there a way to submit a feature request for `VDSUploader.sh` as well? If so what's the best channel? We've been exploring both tools for loading bulk data to OSDU, and we're seeing some gaps e.g. in S3 auth (only profile/dedicated role auth flow available), and no control over the dataset name when loading to SDMS, it generates random name, which prevents from targeting previously created dataset.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/169SEGYImport documentation incomplete/inconsistent (legal values, default values)2023-01-23T12:28:35ZAlexander JaustSEGYImport documentation incomplete/inconsistent (legal values, default values)The comments here refer to the documentation of `SEGYImport` and the deep dive. I found some things that are inconsistent or easy to overlook depending on how one works with the tools provided by OpenVDS.
## CLI help
I find that docume...The comments here refer to the documentation of `SEGYImport` and the deep dive. I found some things that are inconsistent or easy to overlook depending on how one works with the tools provided by OpenVDS.
## CLI help
I find that documentation of `SEGYImport` in the terminal is lacking important information. I tend to use the terminal a lot and thus also use `SEGYImport --help` frequently.
The documentation of valid input values is a bit inconsistent. For some options (attribute name, attribute unit...) the output shows the accepted options.
```
...
--attribute-name <string>
The name of the primary VDS channel. The
name may be Amplitude (default), Attribute,
Depth, Probability, Time, Vavg, Vint, or
Vrms (default: Amplitude)
--attribute-unit <string>
The units of the primary VDS channel. The
unit name may be blank (default), ft, ft/s,
Hz, m, m/s, ms, or s
...
```
However, for "brick size" and "level of detail" levels the limits are not mentioned:
```
...
-b, --brick-size <value> The brick size for the volume data store.
--lod-levels <value> The number of LODs to generate.
...
```
When digging a bit deeper in the, [developer documentation](https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/cppdoc/namespace/namespaceOpenVDS.html#_CPPv4N7OpenVDS26VolumeDataLayoutDescriptor9BrickSizeE) might even give the expectation that one should be allowed to use larger brick sizes that currently allowed. I assume that brick sizes >256 are only useful for 2D datasets. For level of detail levels, it is explained in the deep dive that the value can be at at least 0 and at most 12.
Additionally to that, neither does the [documentation](https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/vds/deepdive/deepdive.html#brick-size) directly mention that the brick size has to be (certain) powers of two.
The output neither states the default value of the number of LODs. I expected that it would be zero as there is no general recommendation for number of levels in the deep dive (see also comment below).
The output of `SEGYImport --help` also deviates slightly from the [documentation on the homepage](https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/tools/SEGYImport/README.html). I am not sure if it would be possible to synchonize the content of the README with the actual `SEGYImport` output.
There might be further options with incomplete documentation.
## Deep dive
### Margin
It is unclear to me what the correct margin size to chose is and what the actual default is without looking into the source code of `SEGYImport`. In the [deep dive](https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/vds/deepdive/deepdive.html#margin-size) it is mentioned that a margin of 4 should be used as default and that the value is important when working with level of details. No connection to wavelets is mentioned.
In the [README](https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/tools/SEGYImport/README.html) it mentions that the default value for the margin is 0 and 4 if one uses wavelet compression. Checking the source code confirms the statement of the README.
### Level of detail
It is explained when one should use level of details, but no general recommendation for how many level of detail levels one should generate is given (except if one wants to use FAST). However, `SEGYImport` chooses 2 as general default and 4 for poststack data.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/168SEGYImport allows nonsensical values for margin2023-05-05T14:52:33ZAlexander JaustSEGYImport allows nonsensical values for marginI have been playing around a bit with `SEGYImport` and observed the following behavior which does not really make sense to me. I wonder if one could check for such nonsensical values when using `SEGYImport`
## Observations
- I can spec...I have been playing around a bit with `SEGYImport` and observed the following behavior which does not really make sense to me. I wonder if one could check for such nonsensical values when using `SEGYImport`
## Observations
- I can specify negative margin sizes
```
SEGYImport --lod-levels=0 --margin -1 volve.sgy --vdsfile=volve.vds
```
The conversion will go up to `99.X %` and then hang. I did not wait too long and aborted the conversion. The resulting file still looks somewhat reasonable and the header is intact, i.e., I can check the header with `VDSInfo`. The resulting file also looks reasonably large, but a bit smaller than for `--margin 0`. I did not try to run any actual operations (requesting data or similar) from the VDS dataset.
- I can specify margins greater and equal to the brick size.
```
SEGYImport --lod-levels=0 --margin 64 volve.sgy --vdsfile=volve.vds
```
This returns really quickly **without any error message**. However, the resulting VDS dataset is only 2MB large. It seems that (only) the header is written (correctly), but no other content. It have not checked the generated VDS any further though.
For margin sizes greater than the brick size, I wonder what the meaning would be. Should the margin contain container information of neighboring bricks and their neighbors?
- I observe the behavior of generating VDS datasets of 2MB (point above) as soon as my margin is at least half the brick size. In this case 32.
It could be that some of the behavior is linked to the amount of memory (32GB) or storage (about 640 Gi available).
## Platform
- Apple Arm M1 Max
- MacOS 13.1
- OpenVDS 3.0.3 (compile from source)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/167Abort trap 6 on OpenVDS::Open2023-08-29T14:42:46ZErlend HårstadAbort trap 6 on OpenVDS::OpenPassing `OpenVDS::Open` a sas token where the Signed Resource Type (`srt`) parameter contains 'Container' (`c`) and _not_ object (`o`) cause an Abort trap 6. As far as I can tell, this happen for any combination of `srt` options as long ...Passing `OpenVDS::Open` a sas token where the Signed Resource Type (`srt`) parameter contains 'Container' (`c`) and _not_ object (`o`) cause an Abort trap 6. As far as I can tell, this happen for any combination of `srt` options as long as it contains `c` and not `o`. Although this particular combination doesn't make a whole lot of sense when working with VDS, it's still a completely valid sas.
I've only tested this on 3.0.3https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/166Cannot import openvds with Python 3.112023-01-03T12:43:01ZDavid WadeCannot import openvds with Python 3.11There is definitely something very wrong when attempting to import openvds on Python 3.11
Please fix?
```
Collecting openvds
Downloading openvds-3.1.3-cp311-cp311-win_amd64.whl (19.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ...There is definitely something very wrong when attempting to import openvds on Python 3.11
Please fix?
```
Collecting openvds
Downloading openvds-3.1.3-cp311-cp311-win_amd64.whl (19.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.1/19.1 MB 2.3 MB/s eta 0:00:00
Installing collected packages: numpy, openvds
Successfully installed numpy-1.24.1 openvds-3.1.3
(vds-311) PS C:\xxxx\xxxxx\xxxxx> python
Python 3.11.1 (tags/v3.11.1:a7a450f, Dec 6 2022, 19:58:39) [MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import openvds
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\xxxx\xxxx\vds-311\Lib\site-packages\openvds\__init__.py", line 1, in <module>
from .api import *
File "C:\xxxx\xxxx\vds-311\Lib\site-packages\openvds\api.py", line 18, in <module>
import openvds.core
ModuleNotFoundError: No module named 'openvds.core'
```https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/165Cpython3.11 wheels for linux2022-12-22T08:52:47ZFilip BrzękCpython3.11 wheels for linuxCurrently, the PyPI index has 3.1.2 wheels built for cpython3.11 only for windows, AFAIK latest `manylinux_2014` does have cpython3.11.
```log
docker run -it quay.io/pypa/manylinux2014_x86_64 ls /opt/python
cp310-cp310 cp36-cp36m cp38-...Currently, the PyPI index has 3.1.2 wheels built for cpython3.11 only for windows, AFAIK latest `manylinux_2014` does have cpython3.11.
```log
docker run -it quay.io/pypa/manylinux2014_x86_64 ls /opt/python
cp310-cp310 cp36-cp36m cp38-cp38 pp37-pypy37_pp73 pp39-pypy39_pp73
cp311-cp311 cp37-cp37m cp39-cp39 pp38-pypy38_pp73
```
Thanks!https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/164Build fails for OpenSSL v3 due to strict warnings in Azure dependency2022-12-22T08:54:27ZAlexander JaustBuild fails for OpenSSL v3 due to strict warnings in Azure dependencyWhen I want to install OpenVDS in a Docker container using Alpine 3.17 the build fails. This seems to be due to the fact that Alpine has updated their default OpenSSL installation to OpenSSL v3. This triggers warnings in the Azure depend...When I want to install OpenVDS in a Docker container using Alpine 3.17 the build fails. This seems to be due to the fact that Alpine has updated their default OpenSSL installation to OpenSSL v3. This triggers warnings in the Azure dependencies. The dependencies are, by default, compiled with "Warnings as errors".
I am not sure what the best way would be to fix the problem. In general, it would be nice to be able to preinstall the dependency myself and avoid building it during the OpenVDS build process (this applies to all 3rd party dependencies). In that case one could install dependencies beforehand with suitable options and or from the system's package manager. Moreover, one also could reuse the dependencies when upgrading to new OpenVDS versions making the upgrade process faster more and much lighter.
## Further Observations
- On Alpine 3.16 installed OpenSSL v1 so it was no issue back then.
- Setting `DISABLE_STRICT_WARNINGS=OFF` in OpenVDS is not forwarded to the dependency.
## Steps to reproduce
Create a Docker container based on alpine 3.17 and install OpenVDS dependencies
```bash
apk --no-cache add \
curl \
git \
g++ \
gcc \
make \
cmake \
curl-dev \
boost-dev \
libxml2-dev \
libuv-dev \
util-linux-dev
```
Afterwards, download, configure, and build OpenVDS with the following options
```bash
cmake -S . \
-B build \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_JAVA=OFF \
-DBUILD_PYTHON=OFF \
-DBUILD_EXAMPLES=OFF \
-DBUILD_TESTS=OFF \
-DBUILD_DOCS=OFF \
-DDISABLE_AWS_IOMANAGER=ON \
-DDISABLE_AZURESDKFORCPP_IOMANAGER=OFF \
-DDISABLE_GCP_IOMANAGER=ON \
-DDISABLE_DMS_IOMANAGER=OFF \
-DDISABLE_STRICT_WARNINGS=OFF
cmake --build build --config Release --target install --verbose
```
## Error
```text
#11 21.29 -- stderr output is:
[1433](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1434)
#11 21.29 /open-vds/3rdparty/azure-sdk-for-cpp-12.3.0/sdk/core/azure-core/src/cryptography/md5.cpp: In member function 'virtual void {anonymous}::Md5OpenSSL::OnAppend(const uint8_t*, size_t)':
[1434](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1435)
#11 21.29 /open-vds/3rdparty/azure-sdk-for-cpp-12.3.0/sdk/core/azure-core/src/cryptography/md5.cpp:166:65: error: 'int MD5_Update(MD5_CTX*, const void*, size_t)' is deprecated: Since OpenSSL 3.0 [-Werror=deprecated-declarations]
[1435](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1436)
#11 21.29 166 | void OnAppend(const uint8_t* data, size_t length) { MD5_Update(m_context.get(), data, length); }
[1436](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1437)
#11 21.29 | ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[1437](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1438)
#11 21.29 In file included from /open-vds/3rdparty/azure-sdk-for-cpp-12.3.0/sdk/core/azure-core/src/cryptography/md5.cpp:13:
[1438](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1439)
#11 21.29 /usr/include/openssl/md5.h:50:27: note: declared here
[1439](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1440)
#11 21.29 50 | OSSL_DEPRECATEDIN_3_0 int MD5_Update(MD5_CTX *c, const void *data, size_t len);
[1440](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1441)
#11 21.29 | ^~~~~~~~~~
[1441](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1442)
#11 21.29 /open-vds/3rdparty/azure-sdk-for-cpp-12.3.0/sdk/core/azure-core/src/cryptography/md5.cpp: In member function 'virtual std::vector<unsigned char> {anonymous}::Md5OpenSSL::OnFinal(const uint8_t*, size_t)':
[1442](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1443)
#11 21.29 /open-vds/3rdparty/azure-sdk-for-cpp-12.3.0/sdk/core/azure-core/src/cryptography/md5.cpp:172:14: error: 'int MD5_Final(unsigned char*, MD5_CTX*)' is deprecated: Since OpenSSL 3.0 [-Werror=deprecated-declarations]
[1443](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1444)
#11 21.29 172 | MD5_Final(hash, m_context.get());
[1444](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1445)
#11 21.29 | ~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~
[1445](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1446)
#11 21.29 /usr/include/openssl/md5.h:51:27: note: declared here
[1446](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1447)
#11 21.29 51 | OSSL_DEPRECATEDIN_3_0 int MD5_Final(unsigned char *md, MD5_CTX *c);
[1447](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1448)
#11 21.29 | ^~~~~~~~~
[1448](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1449)
#11 21.29 /open-vds/3rdparty/azure-sdk-for-cpp-12.3.0/sdk/core/azure-core/src/cryptography/md5.cpp: In constructor '{anonymous}::Md5OpenSSL::Md5OpenSSL()':
[1449](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1450)
#11 21.29 /open-vds/3rdparty/azure-sdk-for-cpp-12.3.0/sdk/core/azure-core/src/cryptography/md5.cpp:180:13: error: 'int MD5_Init(MD5_CTX*)' is deprecated: Since OpenSSL 3.0 [-Werror=deprecated-declarations]
[1450](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1451)
#11 21.29 180 | MD5_Init(m_context.get());
[1451](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1452)
#11 21.29 | ~~~~~~~~^~~~~~~~~~~~~~~~~
[1452](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1453)
#11 21.29 /usr/include/openssl/md5.h:49:27: note: declared here
[1453](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1454)
#11 21.29 49 | OSSL_DEPRECATEDIN_3_0 int MD5_Init(MD5_CTX *c);
[1454](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1455)
#11 21.29 |
```https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/163Default behaviour of SEGYImport --sample-unit2022-12-15T15:04:09ZErlend HårstadDefault behaviour of SEGYImport --sample-unitHi!
I recently noticed that the default value of SEGYImport --sample-unit is "ms". This seams like a dangerous default. I think it's reasonable to assume that a lot of people (including myself :smile:) will mess up by trusting the defau...Hi!
I recently noticed that the default value of SEGYImport --sample-unit is "ms". This seams like a dangerous default. I think it's reasonable to assume that a lot of people (including myself :smile:) will mess up by trusting the default here. Would a better default be the value of the Trace Value Measurement Unit (byte 203-204) in the SEGY? Or if that might not be trusted and/or rarely present, then another good default could unitless?
I would also _love_ a strict option that fails the creation if something is even slightly off.
Related to axis units, I also have a question about the relationship between axis units and axis name. I would expect that the name and unit corresponded. I.e. if unit = m then name (annotation) = Depth and so forth. However, SEGYImport seams to always write "Sample" as the axis namehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/162Byte swap bug in Ibm2ieee2023-06-20T12:16:12ZArtem ВByte swap bug in Ibm2ieeeWe found that `Ibm2ieee` implementation does not take into account file and host endianness.
For example:
```
template<SEGY::Endianness ENDIANNESS, SEGY::BinaryHeader::DataSampleFormatCode FORMAT>
void copySamples(float * prTarget, con...We found that `Ibm2ieee` implementation does not take into account file and host endianness.
For example:
```
template<SEGY::Endianness ENDIANNESS, SEGY::BinaryHeader::DataSampleFormatCode FORMAT>
void copySamples(float * prTarget, const unsigned char * puSource, int iSampleMin, int iSampleMax)
{
if (ENDIANNESS == SEGY::Endianness::BigEndian)
{
nValue = (int)(puSource[iSample * 4 + 0] << 24 | puSource[iSample * 4 + 1] << 16 | puSource[iSample * 4 + 2] << 8 | puSource[iSample * 4 + 3]);
}
else
{
nValue = (int)(puSource[iSample * 4 + 3] << 24 | puSource[iSample * 4 + 2] << 16 | puSource[iSample * 4 + 1] << 8 | puSource[iSample * 4 + 0]);
}
}
```
use native conversion with real file endianess.
The original 'Ibm2ieee' form SEGY.cpp always swaps bytes:
```
template<SEGY::Endianness ENDIANNESS, SEGY::BinaryHeader::DataSampleFormatCode FORMAT>
void copySamples(float * prTarget, const unsigned char * puSource, int iSampleMin, int iSampleMax)
{
if (FORMAT == SEGY::BinaryHeader::DataSampleFormatCode::IBMFloat)
{
SEGY::Ibm2ieee(prTarget, puSource + iSampleMin * 4, iSampleMax - iSampleMin);
return;
}
}
void ibm2ieee(void * to, const void * from, size_t len)
{
...
#ifdef WIN32
fr = _byteswap_ulong(fr);
#else
fr = __builtin_bswap32(fr);
#endif // WIN32
...
}
```
It is needed to check real file and hosts endianess before swap bytes.
---
It makes wrong traces data for little-endian SEG-Ys with IBMFloats data formathttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/161World coordinates when cdp is not defined2022-11-30T13:39:02ZAlena ChaikouskayaWorld coordinates when cdp is not definedWe believe that this case is unlikely to happen in real files, but wanted to point that out anyway.
When file has no cdp information, asking for information in World coordinates returns data in Annotation coordinate system. If transfor...We believe that this case is unlikely to happen in real files, but wanted to point that out anyway.
When file has no cdp information, asking for information in World coordinates returns data in Annotation coordinate system. If transformation with missing World coordinates is impossible, I would have expected some error message along the way.
SEGY [without_cdp.segy](/uploads/2af802cd954648a22c6a8b03c6241f54/without_cdp.segy):
```
spec.samples = [4, 8]
spec.ilines = [3, 4, 5]
spec.xlines = [10, 11]
DelayRecordingTime: 5,
```
openvds:
```
auto transformer = OpenVDS::IJKCoordinateTransformer(layout);
auto annotation = transformer.IJKIndexToAnnotation({0, 0, 0})
auto world = transformer.IJKIndexToWorld({ 0, 0, 0 });
```
Result:
```
Annotation: 3 10 5
World: 3 10 -5
```
World coordinates created from transformed I/J are same as annotation, which is misleading.
World coordinate created from transformed K is `-Time` (also in files with cdp), which seems strange, though somewhat understandable.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/160Extend documentation of examples and supply sample files2022-11-18T12:56:34ZAlexander JaustExtend documentation of examples and supply sample filesIt is great that there are some examples on how to use OpenVDS. It would be even more helpful if there would be a short explanation of what each example does, what assumptions are made and, if necessary, that input files would be include...It is great that there are some examples on how to use OpenVDS. It would be even more helpful if there would be a short explanation of what each example does, what assumptions are made and, if necessary, that input files would be included. I think that would make life much easier for beginners.
For example, the [npz_to_vds.py](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/blob/master/examples/NpzToVds/npz_to_vds.py) script is really helpful, but it would be nice if the input file would be provided with the script. It is not immediately clear that what assumptions on the file are made. When I looked into the file first I had the following questions:
- Why is there a `--npy` command line parameter that is never used?
- Why do the axis descriptors seem to expect x, y and z to be in a certain range [0,2000]?
- Why is the value range computed in the way it is computed? What is the "correct" way to give a value range? Should it be certain percentiles?
- Where is the input file or how can I create a valid input file myself?
- How does writing data via the page accessor actually work and where do I find more information about that?
- Do I have to use `open` and `close` for interaction with the VDS file or could I also use
```python
...
with openvds.create(
args.url,
args.connection,
layoutDescriptor,
axisDescriptors,
channelDescriptors,
metaData,
compressionMethod,
compressionTolerance
) as vds:
layout = openvds.getLayout(vds)
...
accessor.commit()