Open VDS issueshttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues2022-11-21T09:34:46Zhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/159Online Python documentation incomplete2022-11-21T09:34:46ZAlexander JaustOnline Python documentation incompleteIt seems that the online documentation of the Python interface is incomplete. I did not go through everything, but I found the following inconsistencies and problems.
Would it be possible to expose more documentation on the homepage? I ...It seems that the online documentation of the Python interface is incomplete. I did not go through everything, but I found the following inconsistencies and problems.
Would it be possible to expose more documentation on the homepage? I am not sure if some documentation is simply missing or not generated on purpose. In all cases that I have checked the classes and methods have a documentation accessible via Python's `help(...)` function.
## Observations
### VolumeDataLayoutDescriptor
I do not find anything about the constructors of `class VolumeDataLayoutDescriptor` in the [online documentation](https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/python-api.html#openvds.VolumeDataLayoutDescriptor). When I query the documentation via Python I get much more information including the constructor.
That means if I do the following in a Python session
```text
import openvds
help(openvds.VolumeDataLayoutDescriptor)
```
gives the following (shortened) output
```text
Help on class VolumeDataLayoutDescriptor in module openvds.core:
class VolumeDataLayoutDescriptor(pybind11_builtins.pybind11_object)
| Method resolution order:
| VolumeDataLayoutDescriptor
| pybind11_builtins.pybind11_object
| builtins.object
|
| Methods defined here:
|
| __init__(...)
| __init__(*args, **kwargs)
| Overloaded function.
|
| 1. __init__(self: openvds.core.VolumeDataLayoutDescriptor) -> None
|
| 2. __init__(self: openvds.core.VolumeDataLayoutDescriptor, brickSize: OpenVDS::VolumeDataLayoutDescriptor::BrickSize, negativeMargin: int, positiveMargin: int, brickSize2DMultiplier: int, lodLevels: OpenVDS::VolumeDataLayoutDescriptor::LODLevels, options: OpenVDS::VolumeDataLayoutDescriptor::Options, fullResolutionDimension: int = 0) -> None
|
...
```
As you can see there is some information on the constructor.
### VolumeDataRequest
There are (at least?) two types of `VolumeDataRequest`, but only one is documented in the [online documentation](https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/python-api.html#openvds.VolumeDataRequest).
The documentation explains the usage of an object of type `openvds.core.VolumeDataRequest`, see
```text
>>> import openvds
>>> data_request = openvds.VolumeDataRequest
>>> print(data_request)
<class 'openvds.core.VolumeDataRequest'>
>>> buffer = data_request.buffer
>>> data = data_request.data
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: type object 'openvds.core.VolumeDataRequest' has no attribute 'data'
```
This class does not have any attribute called `data` so the error is correct.
However, a function call to the function `requestVolumeSubset` of a `VolumeDataAccessManager` will return an object of type `openvds.volumedataaccess.VolumeDataRequest` with different interface which has a property called `data` instead of `buffer`.
```text
>>> import openvds
>>> data_request = openvds.volumedataaccess.VolumeDataRequest
>>> print(data_request)
<class 'openvds.volumedataaccess.VolumeDataRequest'>
>>> buffer = data_request.buffer
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: type object 'VolumeDataRequest' has no attribute 'buffer'
>>> data = data_request.data
```
This class does not have a `buffer` attribute so the error message is fine. It is not clear from the documentation of [`VolumeDataAccessManager`](https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/python-api.html#id59) that the return type is not a `openvds.core.VolumeDataRequest`.
### openvds.IJKCoordinateTransformer
There is no documentation on the `IJKCoordinateTransformer` for Python even though it is accessible from Python
```text
>>> transformer = openvds.IJKCoordinateTransformer
>>> print(transformer)
<class 'openvds.core.IJKCoordinateTransformer'>
```
### VolumeDataAccessManager
The type annotation for the constructor of the `handle` seems to be off as it is `int`, but should be `openvds.core.VDS`.
```text
>>> vdam = openvds.VolumeDataAccessManager
>>> print(vdam)
<class 'openvds.volumedataaccess.VolumeDataAccessManager'>
>>> vdam = openvds.VolumeDataAccessManager(1)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/aej/software/openvds-3.0.3-install-python/python/openvds/volumedataaccess.py", line 184, in __init__
self._manager = openvds.core.getAccessManager(handle)
TypeError: getAccessManager(): incompatible function arguments. The following argument types are supported:
1. (handle: openvds.core.VDS) -> OpenVDS::VolumeDataAccessManager
Invoked with: 1
```
Update: In the last case, supplying an object of type `openvds.core.VDS` also fails so I assume the documentation might be off here or I misunderstand something.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/158Installation of master fails on MacOS 13.0 with Arm642024-02-01T07:15:58ZAlexander JaustInstallation of master fails on MacOS 13.0 with Arm64## Description
I am trying to build OpenVDS `master` on an Arm64 Mac with the current MacOS release (Ventura), but it fails. Any input would be appreciated. I would also try to supply patches/merge requests where it makes sense.
I am u...## Description
I am trying to build OpenVDS `master` on an Arm64 Mac with the current MacOS release (Ventura), but it fails. Any input would be appreciated. I would also try to supply patches/merge requests where it makes sense.
I am using the following command to configure
```text
cmake -S . \
-B build \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_SHARED_LIBS=ON \
-DBUILD_JAVA=OFF \
-DBUILD_PYTHON=ON \
-DBUILD_EXAMPLES=ON \
-DBUILD_TESTS=OFF \
-DBUILD_DOCS=OFF \
-DDISABLE_AWS_IOMANAGER=ON \
-DDISABLE_AZURESDKFORCPP_IOMANAGER=ON \
-DDISABLE_GCP_IOMANAGER=ON \
-DDISABLE_DMS_IOMANAGER=ON \
-DDISABLE_STRICT_WARNINGS=ON \
-DCMAKE_INSTALL_PREFIX="${INSTALLATION_DIR}" \
```
where `INSTALLATION_DIR` points to `/Users/aej/software/openvds-master-install-python`.
and the following command line for building OpenVDS
```text
cmake --build "build" \
--config Release \
--target install \
-j 1 \
--verbose \
```
## Expectation
OpenVDS is built and installed in the specified directory.
## Actual behavior
The build fails. I found the following problems
1. If I delete a downloaded third-party dependency from the `3rdParty` directory, delete my build directory and then rerun the CMake configuration step the automatic fetching of the library fails. After a second deletion of the build directory and rerunning the CMake configuration step the third-party library seems to be fetched correctly.
2. I get a problem due to the inclusion of `curl.h` by `cpprestsdk`
```text
In file included from /Users/aej/software/compilescripts/openvds/openvds-3.1.0-src/src/OpenVDS/IO/IOManagerCurl.h:41:
/Library/Developer/CommandLineTools/SDKs/MacOSX13.0.sdk/usr/include/curl/curl.h:115:41: error: too few arguments provided to function-like macro invocation
__has_declspec_attribute(dllimport))
```
This seems to be related to changed behavior of LLVM/clang and it appears [with other projects](https://github.com/llvm/llvm-project/issues/53269) and has been reported to [cURL as well](https://github.com/curl/curl/issues/8293). It seems to be some interaction of cURL and casablanca. There are an [issue](https://github.com/microsoft/cpprestsdk/issues/1710) and a [pull request](https://github.com/microsoft/cpprestsdk/pull/1723) in the `cpprestsdk` repository for this, but they say that this will not be fixed since `cpprestsdk` is in maintenance mode.
I can fix it by commenting out the `#define dllimport`, but I am not sure if that is the best thing to do.
```text
cpprestapi_file="${SOURCE_DIR}/3rdparty/cpprestapi-2.10.16/Release/include/cpprest/details/cpprest_compat.h"
sed -i '' 's/\#define dllimport/\/\/\#define dllimport/' "${cpprestapi_file}"
sed -i '' 's/\/\/\/\/\#define dllimport/\/\/\#define dllimport/' "${cpprestapi_file}"
```
As the [`cpprestsdk`](https://github.com/microsoft/cpprestsdk) project is marked as being in maintenance mode so maybe it is necessary to move to another project in the (near?) future.
Side question: Why is the package called `cpprestapi` within the OpenVDS project? It makes debugging a bit confusing since the actual package/repository is called `cpprestsdk`.
3. Building the AWS IOManager fails
```text
/Users/aej/software/compilescripts/openvds/openvds-master-src/src/OpenVDS/IO/IOManagerAWSCurl.h:10:10: fatal error: 'aws/crt/auth/Credentials.h' file not found
#include <aws/crt/auth/Credentials.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
```
The file exists though if I search for in from the OpenVDS repository root
```text
$ find . -iname "Credentials.h" -type f
./3rdparty/google-cloud-cpp-1.14.0/google/cloud/storage/oauth2/credentials.h
./3rdparty/aws-cpp-sdk-1.9.336_/aws-cpp-sdk-cognito-identity/include/aws/cognito-identity/model/Credentials.h
./3rdparty/aws-cpp-sdk-1.9.336_/aws-cpp-sdk-finspace-data/include/aws/finspace-data/model/Credentials.h
./3rdparty/aws-cpp-sdk-1.9.336_/aws-cpp-sdk-connect/include/aws/connect/model/Credentials.h
./3rdparty/aws-cpp-sdk-1.9.336_/aws-cpp-sdk-sts/include/aws/sts/model/Credentials.h
./3rdparty/aws-cpp-sdk-1.9.336_/crt/aws-crt-cpp/include/aws/crt/auth/Credentials.h
./3rdparty/aws-cpp-sdk-1.9.336_/crt/aws-crt-cpp/crt/aws-c-auth/include/aws/auth/credentials.h
```
I am currently a bit stuck at this step since I did not find any straightforward way yet to avoid this problem. I assume that the include paths is not populated properly.
## System
- Arm64 MacOS 13.0.1
- OpenVDS `master` branch
- clang 14.0.0
```text
$ clang --version
Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: arm64-apple-darwin22.1.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
```
- cmake 3.24.3 (via Homebrew)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/157Behavior for files with irregular inlines/crosslines2022-11-15T17:29:04ZAlena ChaikouskayaBehavior for files with irregular inlines/crosslines_(sorry, I will try to divide issue into parts, as for some reason it gets constantly marked as spam)_
While playing with openvds we accidentally created synthetic segy files which are imported into vds incorrectly (roundtrip breaks).
W..._(sorry, I will try to divide issue into parts, as for some reason it gets constantly marked as spam)_
While playing with openvds we accidentally created synthetic segy files which are imported into vds incorrectly (roundtrip breaks).
We are not sure how likely files like that can appear in reality, but our domain knowledge source tells us that it is possible in theory.
1. File [broken1.segy](/uploads/1e35d0885342952d3f3c5ec69273e02c/broken1.segy) with ilines `[1, 6, 11, 15]`
(Stride is 5, last element is at distance 4)
2. File [broken2.segy](/uploads/f2e69b64faf656cdb771ac2215264221/broken2.segy) with ilines `[1, 6, 11, 16, 21, 26, 27]`
(Stride is 5, last element is at distance 1)
Some rows just get lost, never to be found in vds.
Main observation is that as openvds for some reason on purpose ignores the distance between two last values, we need to supply a different distance to last element to reproduce this behavior.
How short/long is the distance of the last element might also matter as changing [this](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/blob/master/tools/SEGYImport/SEGYImport.cpp#L2185) suspicious piece of code (as it seems that `a + (a-b)%c - b` is not divisible by `c`) fixed only one of those cases for me.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/156SEGYImport run on segy with format codes 3 and 8 creates wrong data2022-11-23T09:30:29ZAlena ChaikouskayaSEGYImport run on segy with format codes 3 and 8 creates wrong dataSEGYImport states to support the following data sample format codes:
```
3 = 2-byte, two's complement integer
8 = 1-byte, two's complement integer
```
Yet the conversion results seem wrong.
Attached are the synthetic files with 3 iline...SEGYImport states to support the following data sample format codes:
```
3 = 2-byte, two's complement integer
8 = 1-byte, two's complement integer
```
Yet the conversion results seem wrong.
Attached are the synthetic files with 3 ilines and xlines [100_as_format_3.segy](/uploads/a03cb15e40a745ce15f23797d4765815/100_as_format_3.segy)[100_as_format_8.segy](/uploads/14721b426d8805a3140f5c877a15ab1f/100_as_format_8.segy).
Formats are 3 and 8.
All the values in the files equal to 100. [One can see that data is stored as `00 64` (format 3) and `64` (format 8) which correspond to two's complement integer notation. Processing tools also read the data as `100`.]
Running
```
SEGYImport --vdsfile 100_as_format_8.vds 100_as_format_8.segy
SEGYImport --vdsfile 100_as_format_3.vds 100_as_format_3.segy
```
creates vds files with corresponding formats:
```
"format" : "Format_U16"
"format" : "Format_U8"
```
Running modified code from one of the examples
```
sliceDimension=2
sliceIndex=0
accessManager = openvds.getAccessManager(vds)
layout = openvds.getLayout(vds)
axisDescriptors = [layout.getAxisDescriptor(dim) for dim in range(layout.getDimensionality())]
min = tuple(sliceIndex + 0 if dim == sliceDimension else 0 for dim in range(6))
max = tuple(sliceIndex + 1 if dim == sliceDimension else layout.getDimensionNumSamples(dim) for dim in range(6))
req = accessManager.requestVolumeSubset(min, max, format = format)
height = max[0] if sliceDimension != 0 else max[1]
width = max[2] if sliceDimension != 2 else max[1]
data = req.data.reshape(width, height).transpose()
print(data)
```
with
```
vds=openvds.open("100_as_format_3.vds")
format = openvds.VolumeDataChannelDescriptor.Format.Format_U16
```
gives a slice of `32868`
and with
```
vds=openvds.open("100_as_format_8.vds")
format = openvds.VolumeDataChannelDescriptor.Format.Format_U8
```
gives a slice of `228`
which are not `100`.
This seems to be caused by code [adding 32768 and 128](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/blob/master/src/SEGYUtils/VDSSEGYInfo.cpp#L7) to the values respectively, but I fail to see why it was there in the first place.
All other supported format codes in spot-tests returned the expected values (check was run on little-endian only system/files).
Is there something I misunderstand about these two codes?https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/155Allocation error when using openvds.whl2023-03-28T14:58:41ZSubha RamAllocation error when using openvds.whlCurrently trying to read a .vds from SDMS using vds = openvds.open(URL,connection_string).
Get the following error:
RuntimeError: Error on downloading VolumeDataLayout object: bad allocation.
Any suggestions ?Currently trying to read a .vds from SDMS using vds = openvds.open(URL,connection_string).
Get the following error:
RuntimeError: Error on downloading VolumeDataLayout object: bad allocation.
Any suggestions ?https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/154Crash in Segy import while eject flash drive with segy file (or disconnect la...2023-03-28T15:10:48ZArtem ВCrash in Segy import while eject flash drive with segy file (or disconnect lan/wi-fi with source segy file)It is extremely hard to reproduce, but bug is already exists in all releases of open-vds.
Let's discusse two different cases:
1. Move source segy file to flash drive
2. Import segy file (via SEGImport utils)
3. Eject flash drive while p...It is extremely hard to reproduce, but bug is already exists in all releases of open-vds.
Let's discusse two different cases:
1. Move source segy file to flash drive
2. Import segy file (via SEGImport utils)
3. Eject flash drive while processing
4. Crash without any dump
Another case:
1. Move source segy file to external network drive (google drive, smb or any other)
2. Import segy file (via SEGImport utils)
3. Unplug ethernet cable (disconnect from wifi/lan, close your corporate vpn connection or any other) while processing
4. Crash without any dump
I found, that main reason of crash inside `TraceDataManager`
Crash will be here:
```
const char* header = traceDataManager.getTraceData(trace, error);
if (error.code)
{
outputPrinter.printWarning("IO", "Failed when reading data", fmt::format("{}", error.code), error.string);
break;
}
const void* data = header + SEGY::TraceHeaderSize;
int primaryTest = fileInfo.Is2D() ? 0 : SEGY::ReadFieldFromHeader(header, fileInfo.m_primaryKey, fileInfo.m_headerEndianness),
secondaryTest;
```
![image](/uploads/6b6a1d4b0140865730491d87a35c71da/image.png)
As you can see `TraceDataManager::getTraceData` imp is safe.
```
const char
* basePtr = static_cast<const char *>(pageView->Pointer(error));
```
but... You loose nullptr check while return address.
I added required comparator for MVP:
![image](/uploads/a31740e7589fad71c1f61cea788424e4/image.png)
And yes - `basePtr` is nullptr.
Return ptr with any offset will crash application.
---
Solution: Add code snippet
```
const char *
getTraceData(int64_t traceNumber, OpenVDS::Error & error) const
{
...
// Additional check
if (basePtr == nullptr)
{
error.code = 1;
error.string = "Failed to acquire pageView pointer";
return nullptr;
}
return basePtr + (traceNumber - pageTrace) * m_traceByteSize;
}
```
---
How to 100% reproduce:
0. Start SEGYImport with debug session
1. Copy segy to flash, start importing and waith for 10-20-N% done.
2. Set breakpoint to line: ``` for (int64_t trace(firstTrace); trace <= segment->m_traceStop && error.code == 0; ++trace, ++tertiaryIndex)```
3. Eject flash drive
4. Iterate over loop up to crash :)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/153Cannot create IJKCoordinateTransformer for 2D dataset (Python)2022-10-31T14:31:13ZAlexander JaustCannot create IJKCoordinateTransformer for 2D dataset (Python)## Description
I am currently playing around with the creation of VDS files. I am especially interested in working via the Python interface and the different coordinate systems. I set up the a small [Python script](/uploads/8d79d553af1f...## Description
I am currently playing around with the creation of VDS files. I am especially interested in working via the Python interface and the different coordinate systems. I set up the a small [Python script](/uploads/8d79d553af1f908973720a1e03b21e06/write_2d_vds_data_testing.py) that creates a simple 2D dataset from a NumPy array with random content. Parts of the script is based on the `npz_to_vds.py` script from the examples. I would like to convert between inline/crossline coordinates, voxel coordinates and world coordinates.
In my script, the creation of the VDS file is successful. I also see that the file is recognized as 2D file by OpenVDS during writing since the chunks written to the page buffer are 4*brick_size. However, when I want to obtain the `IJKCoordinateTransformer` for this file, I run into the following exception
```text
Exception:
Dimension -1 is not a valid dimension. Dimensionality_Max is 6.
```
When I create a 3D file with only one coordinate in z direction, obtaining the transformer seems to be successful.
## Expectation
I obtain the coordinate transformer which allows me to transform between [different coordinate](https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/cppdoc/struct/structOpenVDS_1_1IJKCoordinateTransformer.html) systems (ijk, inline/crossline etc.).
## Questions
- Is this behavior expected? I assumed that I could still work with the IJK transformer.
- Do I create the file as a 2D file in a wrong way?
- If the behavior is expected, would it be possible to extend the transformer to work with 2D data and/or as a quick fix to make the error message more expressive?
## System
- Arm64 MacOS 12.6
- VDS 3.0.3 with Python interfacehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/152Broken Win ErrorToString implementation for non-english system lang2023-04-04T19:09:53ZArtem ВBroken Win ErrorToString implementation for non-english system langLet's explain minimum code sample:
```
OpenVDS::Error error;
OpenVDS::SetIoError(8,error );
```
We want to get description with system lang on Windows.
```
error {code=8 string="???????????? ???????? ?????? ??? ????????? ???? ??...Let's explain minimum code sample:
```
OpenVDS::Error error;
OpenVDS::SetIoError(8,error );
```
We want to get description with system lang on Windows.
```
error {code=8 string="???????????? ???????? ?????? ??? ????????? ???? ???????.\r\n" } OpenVDS::VDSError
```
*Non-english system lang* will return unreadable description. For example - I use Kazakh lang.
Solution:
```
std::string ws2s(const std::wstring& s)
{
int len;
int slength = (int)s.length() + 1;
len = WideCharToMultiByte(CP_UTF8, 0, s.c_str(), slength, 0, 0, 0, 0);
std::string r(len, '\0');
WideCharToMultiByte(CP_UTF8, 0, s.c_str(), slength, &r[0], len, 0, 0);
return r;
}
std::string ErrorToString(DWORD error)
{
wchar_t buf[256];
FormatMessageW(FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,
NULL, error, MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT),
buf, (sizeof(buf) / sizeof(wchar_t)), NULL);
std::wstring ws(&buf[0]);
return ws2s(ws);
}
```
Or any equivalent of wchar usage in FormatMessage WinAPI call.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/1513.0.4 fmt related build failure in IOManagerAzureSdkForCpp2022-11-01T11:40:00ZAlena Chaikouskaya3.0.4 fmt related build failure in IOManagerAzureSdkForCppHi,
We have a problem with building 3.0.4 with new azure sdk.
We are running
```
RUN cmake -S . \
-B build \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_JAVA=OFF \
-DBUILD_PYTHON=OFF \
-DBUILD_EXAMPLES=OFF \
-DBUILD_TE...Hi,
We have a problem with building 3.0.4 with new azure sdk.
We are running
```
RUN cmake -S . \
-B build \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_JAVA=OFF \
-DBUILD_PYTHON=OFF \
-DBUILD_EXAMPLES=OFF \
-DBUILD_TESTS=OFF \
-DBUILD_DOCS=OFF \
-DDISABLE_AWS_IOMANAGER=ON \
-DDISABLE_AZURESDKFORCPP_IOMANAGER=OFF \
-DDISABLE_GCP_IOMANAGER=ON \
-DDISABLE_DMS_IOMANAGER=OFF \
-DDISABLE_STRICT_WARNINGS=OFF
```
Unfortunately on all checked OS (alpine, etc) that fails with
```
In file included from /open-vds/3rdparty/fmt-9.1.0/include/fmt/format.h:48,
from /open-vds/src/OpenVDS/IO/IOManagerAzureSdkForCpp.h:33,
from /open-vds/src/OpenVDS/IO/IOManager.cpp:32:
/open-vds/3rdparty/fmt-9.1.0/include/fmt/core.h:1334:47: error: there are no arguments to 'format_as' that depend on a template parameter, so a declaration of 'format_as' must be available [-fpermissive]
```
If in the build script we exchange 9.1.0 with 7.1.3, build is run fine.
But then on usage new errors appear:
```
libazure-core.so, needed by /open-vds/Dist/OpenVDS/lib/libopenvds.so, not found (try using -rpath or -rpath-link)
libazure-storage-blobs.so, needed by /open-vds/Dist/OpenVDS/lib/libopenvds.so, not found (try using -rpath or -rpath-link)
libazure-storage-common.so, needed by /open-vds/Dist/OpenVDS/lib/libopenvds.so, not found (try using -rpath or -rpath-link)
```
That makes sense as only some azure libraries are copied:
```
Installing: /open-vds/Dist/OpenVDS/lib/libazurestorage.so.7.5
Installing: /open-vds/Dist/OpenVDS/lib/libazurestorage.so.7
```
when before (code based on version 2.3.3) `libazure-core.so`, `libazure-storage-blobs.so` and `libazure-storage-common.so` were installed as well.
```
Installing: /open-vds/Dist/OpenVDS/lib/libazurestorage.so.7.5
Installing: /open-vds/Dist/OpenVDS/lib/libazurestorage.so.7
Installing: /open-vds/Dist/OpenVDS/lib/libazurestorage.so
Installing: /open-vds/Dist/OpenVDS/lib/libazure-core.so
Installing: /open-vds/Dist/OpenVDS/lib/libazure-storage-common.so
Installing: /open-vds/Dist/OpenVDS/lib/libazure-template.so
Installing: /open-vds/Dist/OpenVDS/lib/libazure-identity.so
Installing: /open-vds/Dist/OpenVDS/lib/libazure-security-keyvault-common.so
Installing: /open-vds/Dist/OpenVDS/lib/libazure-security-keyvault-keys.so
Installing: /open-vds/Dist/OpenVDS/lib/libazure-storage-blobs.so
Installing: /open-vds/Dist/OpenVDS/lib/libazure-storage-files-datalake.so
Installing: /open-vds/Dist/OpenVDS/lib/libazure-storage-files-shares.so
```
Now to work around this we have to manually copy `azure-sdk-for-cpp_12.3.0_install` artifacts to installation directory.
Can this be fixed?https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/150Create pkg-config configuration (.pc file) for consumption by other projects2022-10-28T07:22:45ZAlexander JaustCreate pkg-config configuration (.pc file) for consumption by other projectsIt would be nice if OpenVDS could generate a [pkg-config](https://www.freedesktop.org/wiki/Software/pkg-config/) configuration file (`.pc` file) for easy consumption of the OpenVDS project in other projects. This would especially help wi...It would be nice if OpenVDS could generate a [pkg-config](https://www.freedesktop.org/wiki/Software/pkg-config/) configuration file (`.pc` file) for easy consumption of the OpenVDS project in other projects. This would especially help with projects that are not using CMake as build system. One example would be projects using Go and its build system. Go has explicit support for pkg-config in the [CGO package](https://pkg.go.dev/cmd/cgo#hdr-Using_cgo_with_the_go_command) to obtain the needed flags for compiling and linking of applications to other packages. Would this be something one could add?
I made a quick test with a minimal template for the pkg-config configuration file which worked in my setup. I have attached the patch. It creates a `openvds.pc` file from the `openvds.pc.in` template file stored in `CMake` via a `configure_file` step in the CMake build process. The configuration file is installed into `TARGET_DIR/lib/pkgconfig/`. If the patch would be already acceptable or only small extra work, I would submit it as merge request.
[0001-Add-pkg-config-configuration-file-generation.patch](/uploads/612c2ede69944f24f2e8cb4ec83d8927/0001-Add-pkg-config-configuration-file-generation.patch)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/149unable to install openvds on amazon linux2022-10-21T12:32:18Zsangamesh hooliunable to install openvds on amazon linuxERROR: Could not find a version that satisfies the requirement openvds (from versions: none)
ERROR: No matching distribution found for openvds
I have tried with all the options mentioned in https://packaging.python.org/en/latest/tutoria...ERROR: Could not find a version that satisfies the requirement openvds (from versions: none)
ERROR: No matching distribution found for openvds
I have tried with all the options mentioned in https://packaging.python.org/en/latest/tutorials/installing-packages/#installing-from-pypihttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/148Open local vds file with 260+ symbols in path2022-10-17T09:54:36ZArtem ВOpen local vds file with 260+ symbols in pathProblem in opening vds file with 260+ symbols path.
For example:
`D:\Documents\Documents\1234567890 168e79156-f70c-41a9-9b84-89f5fb9402e9 8e79156-f70c-41a9-9b84-89f5fb9402e9 8e79156-f70c-41a9-9b84-89f5fb9402e9 8e79156-f70c-41a9-9b84-89f...Problem in opening vds file with 260+ symbols path.
For example:
`D:\Documents\Documents\1234567890 168e79156-f70c-41a9-9b84-89f5fb9402e9 8e79156-f70c-41a9-9b84-89f5fb9402e9 8e79156-f70c-41a9-9b84-89f5fb9402e9 8e79156-f70c-41a9-9b84-89f5fb9402e9 8e79156-\Test_Lateral.StarSteer.data\sc\f1c720af-8e57-4137-874b-11f96e47bda7.vds`
can't be opened. Error:
```
{code=3 string="File::Open: The system cannot find the path specified.\r\n" }
```
YES, 260 Character Path Limit already fixed on my PC :smile:
```
bool File::Open(const std::string& filename, bool isCreate, bool isDestroyExisting, bool isWriteAccess, Error& error)
std::wstring native_name;
s2ws(_cFileName, native_name);
_pxPlatformHandleRead = CreateFileW(
native_name.c_str(),
GENERIC_READ | (isWriteAccess ? GENERIC_WRITE : 0),
FILE_SHARE_READ | FILE_SHARE_WRITE,
NULL,
dwCreationDisposition,
FILE_ATTRIBUTE_NORMAL | FILE_FLAG_RANDOM_ACCESS | FILE_FLAG_OVERLAPPED,
NULL);
}
```
generates `INVALID_HANDLE_VALUE`.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/147Nearest interpolation does not choose the numerically nearest voxel2023-10-19T18:24:41ZErlend HårstadNearest interpolation does not choose the numerically nearest voxel
Hi,
Why does vds’ nearest interpolation interpolate in the range `[0, 1)` in around voxels, and not `[-0.5, 0.5)`? I find this a bit surprising. If I ask for voxel 1.99 in some dimension I would expect to get voxel 2 not 1, as 2 is nume...
Hi,
Why does vds’ nearest interpolation interpolate in the range `[0, 1)` in around voxels, and not `[-0.5, 0.5)`? I find this a bit surprising. If I ask for voxel 1.99 in some dimension I would expect to get voxel 2 not 1, as 2 is numerically “nearer” to 1.99 than 1https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/146SEGYImport includes sas-token in VolumeDataLayout2022-10-17T09:58:21ZErlend HårstadSEGYImport includes sas-token in VolumeDataLayoutHi,
If invoking SEGYImport (v2.3.3) with a blob sas url, the sas is included as part of the metadata in VolumeDataLayout. E.g:
```
SEGYImport --url "azureSAS://myaccount.blob.core.windows.net/container" --url-connection "Suffix=?$SAS"...Hi,
If invoking SEGYImport (v2.3.3) with a blob sas url, the sas is included as part of the metadata in VolumeDataLayout. E.g:
```
SEGYImport --url "azureSAS://myaccount.blob.core.windows.net/container" --url-connection "Suffix=?$SAS" --compression-method=None "https://myaccount.blob.core.windows.net/container/path/mysegy.segy?sp=r&st=2022-09-30T09%3A32%3A03Z&se=2022-09-30T17%3A32%3A03Z&spr=https&sv=2021-06-08&sr=b&sig=dy%2FVXSzS15yjFgMuk0rSYzh2eIXRdjHf0MyaxxLRgI%3D"
```
produces the following entries in the VolumeDataLayout
```
{
"category" : "ImportInformation",
"name" : "DisplayName",
"type" : "String",
"value" : "mysegy.segy?sp=r&st=2022-09-30T09%3A32%3A03Z&se=2022-09-30T17%3A32%3A03Z&spr=https&sv=2021-06-08&sr=b&sig=dy%2FVXSzS15yjFgMuk0rSYzh2eIXRdjHf0MyaxxLRgI%3D"
},
{
"category" : "ImportInformation",
"name" : "InputFileName",
"type" : "String",
"value" : "mysegy.segy?sp=r&st=2022-09-30T09%3A32%3A03Z&se=2022-09-30T17%3A32%3A03Z&spr=https&sv=2021-06-08&sr=b&sig=dy%2FVXSzS15yjFgMuk0rSYzh2eIXRdjHf0MyaxxLRgI%3D"
},
```https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/145Wrong error message in OpenVDSInterfaceImpl::Close2022-09-26T14:42:45ZAnatoly YanchevskyWrong error message in OpenVDSInterfaceImpl::CloseIn function `OpenVDSInterfaceImpl::Close`:
`throw InvalidOperation(fmt::format("Flush in Close failed: {}", error.string).c_str());`
need to change on:
`throw InvalidOperation(fmt::format("Flush in Close failed: {}", flashError.string...In function `OpenVDSInterfaceImpl::Close`:
`throw InvalidOperation(fmt::format("Flush in Close failed: {}", error.string).c_str());`
need to change on:
`throw InvalidOperation(fmt::format("Flush in Close failed: {}", flashError.string).c_str());`https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/144Build OpenVDS+ with the latest msvc 20222022-09-26T09:44:53ZArtem ВBuild OpenVDS+ with the latest msvc 2022Can you provide information about msvc complier version of openvds+ (2.4.3) build?
`msvc_140` - it is actual msvc toolset, isn't it?
Do you have any plans to migrate to newly msvc (2022) and update dlls?Can you provide information about msvc complier version of openvds+ (2.4.3) build?
`msvc_140` - it is actual msvc toolset, isn't it?
Do you have any plans to migrate to newly msvc (2022) and update dlls?https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/143No README.md for HueBDS2022-09-13T12:25:25ZMorten OfstadNo README.md for HueBDSThere is no README.md for the HueBDS tool and it doesn't get an entry in the documentation.There is no README.md for the HueBDS tool and it doesn't get an entry in the documentation.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/142Assert when trying to make slice on connection lost2022-08-30T08:30:45ZArtem ВAssert when trying to make slice on connection lostWe catch assertion in OpenVDS in base scenario with remote (S3) requests + lost internet conenction case:
P1. Open handle to remote S3 and make N requests:
```
std::vector<std::array<float, OpenVDS::Dimensionality_Max>> samplesPosition...We catch assertion in OpenVDS in base scenario with remote (S3) requests + lost internet conenction case:
P1. Open handle to remote S3 and make N requests:
```
std::vector<std::array<float, OpenVDS::Dimensionality_Max>> samplesPositions;
// Generate required points for small area
// And fetch via RequestVolumeSamples
auto vdsRequest(accessManager.RequestVolumeSamples(
OpenVDS::Dimensions_012,
0,
channelIndex,
reinterpret_cast<const float(*)[OpenVDS::Dimensionality_Max]>(samplesPositions.data()),
static_cast<int>(samplesPositions.size()),
OpenVDS::InterpolationMethod::Linear));
```
P2. Wrap P1 as std::future list:
```
std::vector<std::future<std::shared_ptr<VolumeDataRequestFloat>>> requests;
for (std::size_t t(0); t < 100; t++)
{
requests.emplace_back(std::async(makeRequest(t));
}
```
P3. Disconnect ethernet cable or turn off wifi connection
```
Expression: m_pendingDownloadRequests.find(chunk) == m_pendingDownloadRequests.end()
For information on how your program can cause an assertion
failure, see the Visual C++ documentation on asserts
(Press Retry to debug the application - JIT must be enabled)
---------------------------
Abort Retry Ignore
---------------------------
```
![image](/uploads/2b8169f2b0246eeadba6daaa02b6443d/image.png)
Looks like library should check chunk existage before access:
```
bool VolumeDataStoreIOManager::PrepareReadChunkImpl(const VolumeDataChunk &chunk, int adaptiveLevel, Error &error)
{
...
// HERE
if (m_pendingDownloadRequests.find(chunk) == m_pendingDownloadRequests.end())
return false
/// Continue
std::string url = CreateUrlForChunk(layerName, chunk.index);
auto transferHandler = std::make_shared<ReadChunkTransfer>(compressionInfo, (metadataManager != nullptr) ? parsedMetadata.CreateChunkMetadata() : std::vector<uint8_t>());
if (isConstantValue)
{
m_pendingDownloadRequests[chunk] = PendingDownloadRequest(transferHandler);
}
else
{
m_pendingDownloadRequests[chunk] = PendingDownloadRequest(m_ioManager->ReadObject(url, transferHandler, ioRange), transferHandler);
}
return true;
}
```https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/141SEGYImport ignores scaling factor for integral samples2024-02-26T20:14:01ZPaal KvammeSEGYImport ignores scaling factor for integral samplesSEGYImport doesn't seem to handle offset/scale when reading Seg-Y files with integral samples. Actually, Seg-Y doesn't allow specifying offset. But it can specify scale. The scale is found in the TRWF field, bytes 169-170, in the trace h...SEGYImport doesn't seem to handle offset/scale when reading Seg-Y files with integral samples. Actually, Seg-Y doesn't allow specifying offset. But it can specify scale. The scale is found in the TRWF field, bytes 169-170, in the trace header. Assuming I understand the spec correctly.
If the TRWF is the same for all traces then the scale factor (2^-TRWF) could easily be stored in the VDS metadata. So, this is arguably a bug. See createChannelDescriptors() in SEGYImport.cpp.
Varying TRWF is trickier and would probably require the samples to be converted to float with each trace being scaled individually. Note that I have not seen such files in the wild. So, this second issue might be of academic interest only.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/140I have tried to do VDS copy of file around 100GB got below error2022-08-25T09:50:50Zsangamesh hooliI have tried to do VDS copy of file around 100GB got below errorvds.VDSError: [Chunk: 187, Channel: 0 LOD: 0: Read error: ReadSync::GetOverlappedResult: An unexpected network error occurred.
]
VolumeDataAccessManager destructor: there where upload errors
As per my understanding network error should...vds.VDSError: [Chunk: 187, Channel: 0 LOD: 0: Read error: ReadSync::GetOverlappedResult: An unexpected network error occurred.
]
VolumeDataAccessManager destructor: there where upload errors
As per my understanding network error should not stop VDS copy