Open VDS issueshttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues2023-03-15T08:22:49Zhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/170Allow VDSCopy to overwrite existing SDMS dataset: `--allow-overwrite`2023-03-15T08:22:49ZFilip BrzękAllow VDSCopy to overwrite existing SDMS dataset: `--allow-overwrite`Hello,
first of all, thank you very much for the great `3.1` release, and re-implementation of `IOManagers`, for SDMS, it's been very helpful as `sdapi` was troublesome.
While examining new IOManagers with `OPENVDS_DMS_CURL=1`, I have ...Hello,
first of all, thank you very much for the great `3.1` release, and re-implementation of `IOManagers`, for SDMS, it's been very helpful as `sdapi` was troublesome.
While examining new IOManagers with `OPENVDS_DMS_CURL=1`, I have noticed a change in the behavior, that prevents us from writing bulk trace data (VDS content) into the previously created dataset (with the metadata we require), that doesn't yet have any data loaded.
```shell
OPENVDS_DMS_CURL=1 AWS_REGION=us-east-1 VDSCopy ./data/syntethic_data.wavelet.vds sd://osdu/<test-project>/demo-1
[CURL http respons error 409. Automatic rety https://<REDACTED>/api/seismic-store/v3/dataset/tenant/osdu/subproject/<REDACTED>/dataset/demo-1?path=%2F]
...
[Could not create VDS sd://osdu/<REDACTED>/gsi-demo-1] Seismic dms lock failed: Http error respons: 409 -> https://<REDACTED>/api/seismic-store/v3/dataset/tenant/osdu/subproject/<REDACTED>/dataset/demo-1?path=%2F
- [seismic-store-service] The dataset sd://osdu/<REDACTED>/demo-1 already exists[seismic-store-service]
```
when it's run through old DMS flow (using `sdapi`), it's happy to ignore 409, and proceed to write it, command for the reference, but it's results in `seg-fault` at the end (it was reported here #123)
```
AWS_REGION=us-east-1 VDSCopy ./data/syntethic_data.wavelet.vds sd://osdu/<test-project>/demo-1
```
Is it possible to add `--allow-overwrite` similar to the flag available in `VDSUploader.sh` in HueSpace SDK, which would ignore 409, when the dataset was created previously?
Rationale: we want to have control over how the dataset is created in SDMS, for data-lineage reasons, for which we need to create it ourselves, and then populate sd location with VDS content.
Regards,
Filip
PS. Is there a way to submit a feature request for `VDSUploader.sh` as well? If so what's the best channel? We've been exploring both tools for loading bulk data to OSDU, and we're seeing some gaps e.g. in S3 auth (only profile/dedicated role auth flow available), and no control over the dataset name when loading to SDMS, it generates random name, which prevents from targeting previously created dataset.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/168SEGYImport allows nonsensical values for margin2023-05-05T14:52:33ZAlexander JaustSEGYImport allows nonsensical values for marginI have been playing around a bit with `SEGYImport` and observed the following behavior which does not really make sense to me. I wonder if one could check for such nonsensical values when using `SEGYImport`
## Observations
- I can spec...I have been playing around a bit with `SEGYImport` and observed the following behavior which does not really make sense to me. I wonder if one could check for such nonsensical values when using `SEGYImport`
## Observations
- I can specify negative margin sizes
```
SEGYImport --lod-levels=0 --margin -1 volve.sgy --vdsfile=volve.vds
```
The conversion will go up to `99.X %` and then hang. I did not wait too long and aborted the conversion. The resulting file still looks somewhat reasonable and the header is intact, i.e., I can check the header with `VDSInfo`. The resulting file also looks reasonably large, but a bit smaller than for `--margin 0`. I did not try to run any actual operations (requesting data or similar) from the VDS dataset.
- I can specify margins greater and equal to the brick size.
```
SEGYImport --lod-levels=0 --margin 64 volve.sgy --vdsfile=volve.vds
```
This returns really quickly **without any error message**. However, the resulting VDS dataset is only 2MB large. It seems that (only) the header is written (correctly), but no other content. It have not checked the generated VDS any further though.
For margin sizes greater than the brick size, I wonder what the meaning would be. Should the margin contain container information of neighboring bricks and their neighbors?
- I observe the behavior of generating VDS datasets of 2MB (point above) as soon as my margin is at least half the brick size. In this case 32.
It could be that some of the behavior is linked to the amount of memory (32GB) or storage (about 640 Gi available).
## Platform
- Apple Arm M1 Max
- MacOS 13.1
- OpenVDS 3.0.3 (compile from source)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/167Abort trap 6 on OpenVDS::Open2023-08-29T14:42:46ZErlend HårstadAbort trap 6 on OpenVDS::OpenPassing `OpenVDS::Open` a sas token where the Signed Resource Type (`srt`) parameter contains 'Container' (`c`) and _not_ object (`o`) cause an Abort trap 6. As far as I can tell, this happen for any combination of `srt` options as long ...Passing `OpenVDS::Open` a sas token where the Signed Resource Type (`srt`) parameter contains 'Container' (`c`) and _not_ object (`o`) cause an Abort trap 6. As far as I can tell, this happen for any combination of `srt` options as long as it contains `c` and not `o`. Although this particular combination doesn't make a whole lot of sense when working with VDS, it's still a completely valid sas.
I've only tested this on 3.0.3https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/166Cannot import openvds with Python 3.112023-01-03T12:43:01ZDavid WadeCannot import openvds with Python 3.11There is definitely something very wrong when attempting to import openvds on Python 3.11
Please fix?
```
Collecting openvds
Downloading openvds-3.1.3-cp311-cp311-win_amd64.whl (19.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ...There is definitely something very wrong when attempting to import openvds on Python 3.11
Please fix?
```
Collecting openvds
Downloading openvds-3.1.3-cp311-cp311-win_amd64.whl (19.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.1/19.1 MB 2.3 MB/s eta 0:00:00
Installing collected packages: numpy, openvds
Successfully installed numpy-1.24.1 openvds-3.1.3
(vds-311) PS C:\xxxx\xxxxx\xxxxx> python
Python 3.11.1 (tags/v3.11.1:a7a450f, Dec 6 2022, 19:58:39) [MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import openvds
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\xxxx\xxxx\vds-311\Lib\site-packages\openvds\__init__.py", line 1, in <module>
from .api import *
File "C:\xxxx\xxxx\vds-311\Lib\site-packages\openvds\api.py", line 18, in <module>
import openvds.core
ModuleNotFoundError: No module named 'openvds.core'
```https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/165Cpython3.11 wheels for linux2022-12-22T08:52:47ZFilip BrzękCpython3.11 wheels for linuxCurrently, the PyPI index has 3.1.2 wheels built for cpython3.11 only for windows, AFAIK latest `manylinux_2014` does have cpython3.11.
```log
docker run -it quay.io/pypa/manylinux2014_x86_64 ls /opt/python
cp310-cp310 cp36-cp36m cp38-...Currently, the PyPI index has 3.1.2 wheels built for cpython3.11 only for windows, AFAIK latest `manylinux_2014` does have cpython3.11.
```log
docker run -it quay.io/pypa/manylinux2014_x86_64 ls /opt/python
cp310-cp310 cp36-cp36m cp38-cp38 pp37-pypy37_pp73 pp39-pypy39_pp73
cp311-cp311 cp37-cp37m cp39-cp39 pp38-pypy38_pp73
```
Thanks!https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/164Build fails for OpenSSL v3 due to strict warnings in Azure dependency2022-12-22T08:54:27ZAlexander JaustBuild fails for OpenSSL v3 due to strict warnings in Azure dependencyWhen I want to install OpenVDS in a Docker container using Alpine 3.17 the build fails. This seems to be due to the fact that Alpine has updated their default OpenSSL installation to OpenSSL v3. This triggers warnings in the Azure depend...When I want to install OpenVDS in a Docker container using Alpine 3.17 the build fails. This seems to be due to the fact that Alpine has updated their default OpenSSL installation to OpenSSL v3. This triggers warnings in the Azure dependencies. The dependencies are, by default, compiled with "Warnings as errors".
I am not sure what the best way would be to fix the problem. In general, it would be nice to be able to preinstall the dependency myself and avoid building it during the OpenVDS build process (this applies to all 3rd party dependencies). In that case one could install dependencies beforehand with suitable options and or from the system's package manager. Moreover, one also could reuse the dependencies when upgrading to new OpenVDS versions making the upgrade process faster more and much lighter.
## Further Observations
- On Alpine 3.16 installed OpenSSL v1 so it was no issue back then.
- Setting `DISABLE_STRICT_WARNINGS=OFF` in OpenVDS is not forwarded to the dependency.
## Steps to reproduce
Create a Docker container based on alpine 3.17 and install OpenVDS dependencies
```bash
apk --no-cache add \
curl \
git \
g++ \
gcc \
make \
cmake \
curl-dev \
boost-dev \
libxml2-dev \
libuv-dev \
util-linux-dev
```
Afterwards, download, configure, and build OpenVDS with the following options
```bash
cmake -S . \
-B build \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_JAVA=OFF \
-DBUILD_PYTHON=OFF \
-DBUILD_EXAMPLES=OFF \
-DBUILD_TESTS=OFF \
-DBUILD_DOCS=OFF \
-DDISABLE_AWS_IOMANAGER=ON \
-DDISABLE_AZURESDKFORCPP_IOMANAGER=OFF \
-DDISABLE_GCP_IOMANAGER=ON \
-DDISABLE_DMS_IOMANAGER=OFF \
-DDISABLE_STRICT_WARNINGS=OFF
cmake --build build --config Release --target install --verbose
```
## Error
```text
#11 21.29 -- stderr output is:
[1433](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1434)
#11 21.29 /open-vds/3rdparty/azure-sdk-for-cpp-12.3.0/sdk/core/azure-core/src/cryptography/md5.cpp: In member function 'virtual void {anonymous}::Md5OpenSSL::OnAppend(const uint8_t*, size_t)':
[1434](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1435)
#11 21.29 /open-vds/3rdparty/azure-sdk-for-cpp-12.3.0/sdk/core/azure-core/src/cryptography/md5.cpp:166:65: error: 'int MD5_Update(MD5_CTX*, const void*, size_t)' is deprecated: Since OpenSSL 3.0 [-Werror=deprecated-declarations]
[1435](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1436)
#11 21.29 166 | void OnAppend(const uint8_t* data, size_t length) { MD5_Update(m_context.get(), data, length); }
[1436](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1437)
#11 21.29 | ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[1437](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1438)
#11 21.29 In file included from /open-vds/3rdparty/azure-sdk-for-cpp-12.3.0/sdk/core/azure-core/src/cryptography/md5.cpp:13:
[1438](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1439)
#11 21.29 /usr/include/openssl/md5.h:50:27: note: declared here
[1439](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1440)
#11 21.29 50 | OSSL_DEPRECATEDIN_3_0 int MD5_Update(MD5_CTX *c, const void *data, size_t len);
[1440](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1441)
#11 21.29 | ^~~~~~~~~~
[1441](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1442)
#11 21.29 /open-vds/3rdparty/azure-sdk-for-cpp-12.3.0/sdk/core/azure-core/src/cryptography/md5.cpp: In member function 'virtual std::vector<unsigned char> {anonymous}::Md5OpenSSL::OnFinal(const uint8_t*, size_t)':
[1442](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1443)
#11 21.29 /open-vds/3rdparty/azure-sdk-for-cpp-12.3.0/sdk/core/azure-core/src/cryptography/md5.cpp:172:14: error: 'int MD5_Final(unsigned char*, MD5_CTX*)' is deprecated: Since OpenSSL 3.0 [-Werror=deprecated-declarations]
[1443](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1444)
#11 21.29 172 | MD5_Final(hash, m_context.get());
[1444](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1445)
#11 21.29 | ~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~
[1445](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1446)
#11 21.29 /usr/include/openssl/md5.h:51:27: note: declared here
[1446](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1447)
#11 21.29 51 | OSSL_DEPRECATEDIN_3_0 int MD5_Final(unsigned char *md, MD5_CTX *c);
[1447](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1448)
#11 21.29 | ^~~~~~~~~
[1448](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1449)
#11 21.29 /open-vds/3rdparty/azure-sdk-for-cpp-12.3.0/sdk/core/azure-core/src/cryptography/md5.cpp: In constructor '{anonymous}::Md5OpenSSL::Md5OpenSSL()':
[1449](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1450)
#11 21.29 /open-vds/3rdparty/azure-sdk-for-cpp-12.3.0/sdk/core/azure-core/src/cryptography/md5.cpp:180:13: error: 'int MD5_Init(MD5_CTX*)' is deprecated: Since OpenSSL 3.0 [-Werror=deprecated-declarations]
[1450](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1451)
#11 21.29 180 | MD5_Init(m_context.get());
[1451](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1452)
#11 21.29 | ~~~~~~~~^~~~~~~~~~~~~~~~~
[1452](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1453)
#11 21.29 /usr/include/openssl/md5.h:49:27: note: declared here
[1453](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1454)
#11 21.29 49 | OSSL_DEPRECATEDIN_3_0 int MD5_Init(MD5_CTX *c);
[1454](https://github.com/equinor/vds-slice/actions/runs/3675967242/jobs/6216036291#step:3:1455)
#11 21.29 |
```https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/163Default behaviour of SEGYImport --sample-unit2022-12-15T15:04:09ZErlend HårstadDefault behaviour of SEGYImport --sample-unitHi!
I recently noticed that the default value of SEGYImport --sample-unit is "ms". This seams like a dangerous default. I think it's reasonable to assume that a lot of people (including myself :smile:) will mess up by trusting the defau...Hi!
I recently noticed that the default value of SEGYImport --sample-unit is "ms". This seams like a dangerous default. I think it's reasonable to assume that a lot of people (including myself :smile:) will mess up by trusting the default here. Would a better default be the value of the Trace Value Measurement Unit (byte 203-204) in the SEGY? Or if that might not be trusted and/or rarely present, then another good default could unitless?
I would also _love_ a strict option that fails the creation if something is even slightly off.
Related to axis units, I also have a question about the relationship between axis units and axis name. I would expect that the name and unit corresponded. I.e. if unit = m then name (annotation) = Depth and so forth. However, SEGYImport seams to always write "Sample" as the axis namehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/161World coordinates when cdp is not defined2022-11-30T13:39:02ZAlena ChaikouskayaWorld coordinates when cdp is not definedWe believe that this case is unlikely to happen in real files, but wanted to point that out anyway.
When file has no cdp information, asking for information in World coordinates returns data in Annotation coordinate system. If transfor...We believe that this case is unlikely to happen in real files, but wanted to point that out anyway.
When file has no cdp information, asking for information in World coordinates returns data in Annotation coordinate system. If transformation with missing World coordinates is impossible, I would have expected some error message along the way.
SEGY [without_cdp.segy](/uploads/2af802cd954648a22c6a8b03c6241f54/without_cdp.segy):
```
spec.samples = [4, 8]
spec.ilines = [3, 4, 5]
spec.xlines = [10, 11]
DelayRecordingTime: 5,
```
openvds:
```
auto transformer = OpenVDS::IJKCoordinateTransformer(layout);
auto annotation = transformer.IJKIndexToAnnotation({0, 0, 0})
auto world = transformer.IJKIndexToWorld({ 0, 0, 0 });
```
Result:
```
Annotation: 3 10 5
World: 3 10 -5
```
World coordinates created from transformed I/J are same as annotation, which is misleading.
World coordinate created from transformed K is `-Time` (also in files with cdp), which seems strange, though somewhat understandable.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/158Installation of master fails on MacOS 13.0 with Arm642024-02-01T07:15:58ZAlexander JaustInstallation of master fails on MacOS 13.0 with Arm64## Description
I am trying to build OpenVDS `master` on an Arm64 Mac with the current MacOS release (Ventura), but it fails. Any input would be appreciated. I would also try to supply patches/merge requests where it makes sense.
I am u...## Description
I am trying to build OpenVDS `master` on an Arm64 Mac with the current MacOS release (Ventura), but it fails. Any input would be appreciated. I would also try to supply patches/merge requests where it makes sense.
I am using the following command to configure
```text
cmake -S . \
-B build \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_SHARED_LIBS=ON \
-DBUILD_JAVA=OFF \
-DBUILD_PYTHON=ON \
-DBUILD_EXAMPLES=ON \
-DBUILD_TESTS=OFF \
-DBUILD_DOCS=OFF \
-DDISABLE_AWS_IOMANAGER=ON \
-DDISABLE_AZURESDKFORCPP_IOMANAGER=ON \
-DDISABLE_GCP_IOMANAGER=ON \
-DDISABLE_DMS_IOMANAGER=ON \
-DDISABLE_STRICT_WARNINGS=ON \
-DCMAKE_INSTALL_PREFIX="${INSTALLATION_DIR}" \
```
where `INSTALLATION_DIR` points to `/Users/aej/software/openvds-master-install-python`.
and the following command line for building OpenVDS
```text
cmake --build "build" \
--config Release \
--target install \
-j 1 \
--verbose \
```
## Expectation
OpenVDS is built and installed in the specified directory.
## Actual behavior
The build fails. I found the following problems
1. If I delete a downloaded third-party dependency from the `3rdParty` directory, delete my build directory and then rerun the CMake configuration step the automatic fetching of the library fails. After a second deletion of the build directory and rerunning the CMake configuration step the third-party library seems to be fetched correctly.
2. I get a problem due to the inclusion of `curl.h` by `cpprestsdk`
```text
In file included from /Users/aej/software/compilescripts/openvds/openvds-3.1.0-src/src/OpenVDS/IO/IOManagerCurl.h:41:
/Library/Developer/CommandLineTools/SDKs/MacOSX13.0.sdk/usr/include/curl/curl.h:115:41: error: too few arguments provided to function-like macro invocation
__has_declspec_attribute(dllimport))
```
This seems to be related to changed behavior of LLVM/clang and it appears [with other projects](https://github.com/llvm/llvm-project/issues/53269) and has been reported to [cURL as well](https://github.com/curl/curl/issues/8293). It seems to be some interaction of cURL and casablanca. There are an [issue](https://github.com/microsoft/cpprestsdk/issues/1710) and a [pull request](https://github.com/microsoft/cpprestsdk/pull/1723) in the `cpprestsdk` repository for this, but they say that this will not be fixed since `cpprestsdk` is in maintenance mode.
I can fix it by commenting out the `#define dllimport`, but I am not sure if that is the best thing to do.
```text
cpprestapi_file="${SOURCE_DIR}/3rdparty/cpprestapi-2.10.16/Release/include/cpprest/details/cpprest_compat.h"
sed -i '' 's/\#define dllimport/\/\/\#define dllimport/' "${cpprestapi_file}"
sed -i '' 's/\/\/\/\/\#define dllimport/\/\/\#define dllimport/' "${cpprestapi_file}"
```
As the [`cpprestsdk`](https://github.com/microsoft/cpprestsdk) project is marked as being in maintenance mode so maybe it is necessary to move to another project in the (near?) future.
Side question: Why is the package called `cpprestapi` within the OpenVDS project? It makes debugging a bit confusing since the actual package/repository is called `cpprestsdk`.
3. Building the AWS IOManager fails
```text
/Users/aej/software/compilescripts/openvds/openvds-master-src/src/OpenVDS/IO/IOManagerAWSCurl.h:10:10: fatal error: 'aws/crt/auth/Credentials.h' file not found
#include <aws/crt/auth/Credentials.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
```
The file exists though if I search for in from the OpenVDS repository root
```text
$ find . -iname "Credentials.h" -type f
./3rdparty/google-cloud-cpp-1.14.0/google/cloud/storage/oauth2/credentials.h
./3rdparty/aws-cpp-sdk-1.9.336_/aws-cpp-sdk-cognito-identity/include/aws/cognito-identity/model/Credentials.h
./3rdparty/aws-cpp-sdk-1.9.336_/aws-cpp-sdk-finspace-data/include/aws/finspace-data/model/Credentials.h
./3rdparty/aws-cpp-sdk-1.9.336_/aws-cpp-sdk-connect/include/aws/connect/model/Credentials.h
./3rdparty/aws-cpp-sdk-1.9.336_/aws-cpp-sdk-sts/include/aws/sts/model/Credentials.h
./3rdparty/aws-cpp-sdk-1.9.336_/crt/aws-crt-cpp/include/aws/crt/auth/Credentials.h
./3rdparty/aws-cpp-sdk-1.9.336_/crt/aws-crt-cpp/crt/aws-c-auth/include/aws/auth/credentials.h
```
I am currently a bit stuck at this step since I did not find any straightforward way yet to avoid this problem. I assume that the include paths is not populated properly.
## System
- Arm64 MacOS 13.0.1
- OpenVDS `master` branch
- clang 14.0.0
```text
$ clang --version
Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: arm64-apple-darwin22.1.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
```
- cmake 3.24.3 (via Homebrew)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/156SEGYImport run on segy with format codes 3 and 8 creates wrong data2022-11-23T09:30:29ZAlena ChaikouskayaSEGYImport run on segy with format codes 3 and 8 creates wrong dataSEGYImport states to support the following data sample format codes:
```
3 = 2-byte, two's complement integer
8 = 1-byte, two's complement integer
```
Yet the conversion results seem wrong.
Attached are the synthetic files with 3 iline...SEGYImport states to support the following data sample format codes:
```
3 = 2-byte, two's complement integer
8 = 1-byte, two's complement integer
```
Yet the conversion results seem wrong.
Attached are the synthetic files with 3 ilines and xlines [100_as_format_3.segy](/uploads/a03cb15e40a745ce15f23797d4765815/100_as_format_3.segy)[100_as_format_8.segy](/uploads/14721b426d8805a3140f5c877a15ab1f/100_as_format_8.segy).
Formats are 3 and 8.
All the values in the files equal to 100. [One can see that data is stored as `00 64` (format 3) and `64` (format 8) which correspond to two's complement integer notation. Processing tools also read the data as `100`.]
Running
```
SEGYImport --vdsfile 100_as_format_8.vds 100_as_format_8.segy
SEGYImport --vdsfile 100_as_format_3.vds 100_as_format_3.segy
```
creates vds files with corresponding formats:
```
"format" : "Format_U16"
"format" : "Format_U8"
```
Running modified code from one of the examples
```
sliceDimension=2
sliceIndex=0
accessManager = openvds.getAccessManager(vds)
layout = openvds.getLayout(vds)
axisDescriptors = [layout.getAxisDescriptor(dim) for dim in range(layout.getDimensionality())]
min = tuple(sliceIndex + 0 if dim == sliceDimension else 0 for dim in range(6))
max = tuple(sliceIndex + 1 if dim == sliceDimension else layout.getDimensionNumSamples(dim) for dim in range(6))
req = accessManager.requestVolumeSubset(min, max, format = format)
height = max[0] if sliceDimension != 0 else max[1]
width = max[2] if sliceDimension != 2 else max[1]
data = req.data.reshape(width, height).transpose()
print(data)
```
with
```
vds=openvds.open("100_as_format_3.vds")
format = openvds.VolumeDataChannelDescriptor.Format.Format_U16
```
gives a slice of `32868`
and with
```
vds=openvds.open("100_as_format_8.vds")
format = openvds.VolumeDataChannelDescriptor.Format.Format_U8
```
gives a slice of `228`
which are not `100`.
This seems to be caused by code [adding 32768 and 128](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/blob/master/src/SEGYUtils/VDSSEGYInfo.cpp#L7) to the values respectively, but I fail to see why it was there in the first place.
All other supported format codes in spot-tests returned the expected values (check was run on little-endian only system/files).
Is there something I misunderstand about these two codes?https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/155Allocation error when using openvds.whl2023-03-28T14:58:41ZSubha RamAllocation error when using openvds.whlCurrently trying to read a .vds from SDMS using vds = openvds.open(URL,connection_string).
Get the following error:
RuntimeError: Error on downloading VolumeDataLayout object: bad allocation.
Any suggestions ?Currently trying to read a .vds from SDMS using vds = openvds.open(URL,connection_string).
Get the following error:
RuntimeError: Error on downloading VolumeDataLayout object: bad allocation.
Any suggestions ?https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/154Crash in Segy import while eject flash drive with segy file (or disconnect la...2023-03-28T15:10:48ZArtem ВCrash in Segy import while eject flash drive with segy file (or disconnect lan/wi-fi with source segy file)It is extremely hard to reproduce, but bug is already exists in all releases of open-vds.
Let's discusse two different cases:
1. Move source segy file to flash drive
2. Import segy file (via SEGImport utils)
3. Eject flash drive while p...It is extremely hard to reproduce, but bug is already exists in all releases of open-vds.
Let's discusse two different cases:
1. Move source segy file to flash drive
2. Import segy file (via SEGImport utils)
3. Eject flash drive while processing
4. Crash without any dump
Another case:
1. Move source segy file to external network drive (google drive, smb or any other)
2. Import segy file (via SEGImport utils)
3. Unplug ethernet cable (disconnect from wifi/lan, close your corporate vpn connection or any other) while processing
4. Crash without any dump
I found, that main reason of crash inside `TraceDataManager`
Crash will be here:
```
const char* header = traceDataManager.getTraceData(trace, error);
if (error.code)
{
outputPrinter.printWarning("IO", "Failed when reading data", fmt::format("{}", error.code), error.string);
break;
}
const void* data = header + SEGY::TraceHeaderSize;
int primaryTest = fileInfo.Is2D() ? 0 : SEGY::ReadFieldFromHeader(header, fileInfo.m_primaryKey, fileInfo.m_headerEndianness),
secondaryTest;
```
![image](/uploads/6b6a1d4b0140865730491d87a35c71da/image.png)
As you can see `TraceDataManager::getTraceData` imp is safe.
```
const char
* basePtr = static_cast<const char *>(pageView->Pointer(error));
```
but... You loose nullptr check while return address.
I added required comparator for MVP:
![image](/uploads/a31740e7589fad71c1f61cea788424e4/image.png)
And yes - `basePtr` is nullptr.
Return ptr with any offset will crash application.
---
Solution: Add code snippet
```
const char *
getTraceData(int64_t traceNumber, OpenVDS::Error & error) const
{
...
// Additional check
if (basePtr == nullptr)
{
error.code = 1;
error.string = "Failed to acquire pageView pointer";
return nullptr;
}
return basePtr + (traceNumber - pageTrace) * m_traceByteSize;
}
```
---
How to 100% reproduce:
0. Start SEGYImport with debug session
1. Copy segy to flash, start importing and waith for 10-20-N% done.
2. Set breakpoint to line: ``` for (int64_t trace(firstTrace); trace <= segment->m_traceStop && error.code == 0; ++trace, ++tertiaryIndex)```
3. Eject flash drive
4. Iterate over loop up to crash :)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/152Broken Win ErrorToString implementation for non-english system lang2023-04-04T19:09:53ZArtem ВBroken Win ErrorToString implementation for non-english system langLet's explain minimum code sample:
```
OpenVDS::Error error;
OpenVDS::SetIoError(8,error );
```
We want to get description with system lang on Windows.
```
error {code=8 string="???????????? ???????? ?????? ??? ????????? ???? ??...Let's explain minimum code sample:
```
OpenVDS::Error error;
OpenVDS::SetIoError(8,error );
```
We want to get description with system lang on Windows.
```
error {code=8 string="???????????? ???????? ?????? ??? ????????? ???? ???????.\r\n" } OpenVDS::VDSError
```
*Non-english system lang* will return unreadable description. For example - I use Kazakh lang.
Solution:
```
std::string ws2s(const std::wstring& s)
{
int len;
int slength = (int)s.length() + 1;
len = WideCharToMultiByte(CP_UTF8, 0, s.c_str(), slength, 0, 0, 0, 0);
std::string r(len, '\0');
WideCharToMultiByte(CP_UTF8, 0, s.c_str(), slength, &r[0], len, 0, 0);
return r;
}
std::string ErrorToString(DWORD error)
{
wchar_t buf[256];
FormatMessageW(FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,
NULL, error, MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT),
buf, (sizeof(buf) / sizeof(wchar_t)), NULL);
std::wstring ws(&buf[0]);
return ws2s(ws);
}
```
Or any equivalent of wchar usage in FormatMessage WinAPI call.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/1513.0.4 fmt related build failure in IOManagerAzureSdkForCpp2022-11-01T11:40:00ZAlena Chaikouskaya3.0.4 fmt related build failure in IOManagerAzureSdkForCppHi,
We have a problem with building 3.0.4 with new azure sdk.
We are running
```
RUN cmake -S . \
-B build \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_JAVA=OFF \
-DBUILD_PYTHON=OFF \
-DBUILD_EXAMPLES=OFF \
-DBUILD_TE...Hi,
We have a problem with building 3.0.4 with new azure sdk.
We are running
```
RUN cmake -S . \
-B build \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_JAVA=OFF \
-DBUILD_PYTHON=OFF \
-DBUILD_EXAMPLES=OFF \
-DBUILD_TESTS=OFF \
-DBUILD_DOCS=OFF \
-DDISABLE_AWS_IOMANAGER=ON \
-DDISABLE_AZURESDKFORCPP_IOMANAGER=OFF \
-DDISABLE_GCP_IOMANAGER=ON \
-DDISABLE_DMS_IOMANAGER=OFF \
-DDISABLE_STRICT_WARNINGS=OFF
```
Unfortunately on all checked OS (alpine, etc) that fails with
```
In file included from /open-vds/3rdparty/fmt-9.1.0/include/fmt/format.h:48,
from /open-vds/src/OpenVDS/IO/IOManagerAzureSdkForCpp.h:33,
from /open-vds/src/OpenVDS/IO/IOManager.cpp:32:
/open-vds/3rdparty/fmt-9.1.0/include/fmt/core.h:1334:47: error: there are no arguments to 'format_as' that depend on a template parameter, so a declaration of 'format_as' must be available [-fpermissive]
```
If in the build script we exchange 9.1.0 with 7.1.3, build is run fine.
But then on usage new errors appear:
```
libazure-core.so, needed by /open-vds/Dist/OpenVDS/lib/libopenvds.so, not found (try using -rpath or -rpath-link)
libazure-storage-blobs.so, needed by /open-vds/Dist/OpenVDS/lib/libopenvds.so, not found (try using -rpath or -rpath-link)
libazure-storage-common.so, needed by /open-vds/Dist/OpenVDS/lib/libopenvds.so, not found (try using -rpath or -rpath-link)
```
That makes sense as only some azure libraries are copied:
```
Installing: /open-vds/Dist/OpenVDS/lib/libazurestorage.so.7.5
Installing: /open-vds/Dist/OpenVDS/lib/libazurestorage.so.7
```
when before (code based on version 2.3.3) `libazure-core.so`, `libazure-storage-blobs.so` and `libazure-storage-common.so` were installed as well.
```
Installing: /open-vds/Dist/OpenVDS/lib/libazurestorage.so.7.5
Installing: /open-vds/Dist/OpenVDS/lib/libazurestorage.so.7
Installing: /open-vds/Dist/OpenVDS/lib/libazurestorage.so
Installing: /open-vds/Dist/OpenVDS/lib/libazure-core.so
Installing: /open-vds/Dist/OpenVDS/lib/libazure-storage-common.so
Installing: /open-vds/Dist/OpenVDS/lib/libazure-template.so
Installing: /open-vds/Dist/OpenVDS/lib/libazure-identity.so
Installing: /open-vds/Dist/OpenVDS/lib/libazure-security-keyvault-common.so
Installing: /open-vds/Dist/OpenVDS/lib/libazure-security-keyvault-keys.so
Installing: /open-vds/Dist/OpenVDS/lib/libazure-storage-blobs.so
Installing: /open-vds/Dist/OpenVDS/lib/libazure-storage-files-datalake.so
Installing: /open-vds/Dist/OpenVDS/lib/libazure-storage-files-shares.so
```
Now to work around this we have to manually copy `azure-sdk-for-cpp_12.3.0_install` artifacts to installation directory.
Can this be fixed?https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/150Create pkg-config configuration (.pc file) for consumption by other projects2022-10-28T07:22:45ZAlexander JaustCreate pkg-config configuration (.pc file) for consumption by other projectsIt would be nice if OpenVDS could generate a [pkg-config](https://www.freedesktop.org/wiki/Software/pkg-config/) configuration file (`.pc` file) for easy consumption of the OpenVDS project in other projects. This would especially help wi...It would be nice if OpenVDS could generate a [pkg-config](https://www.freedesktop.org/wiki/Software/pkg-config/) configuration file (`.pc` file) for easy consumption of the OpenVDS project in other projects. This would especially help with projects that are not using CMake as build system. One example would be projects using Go and its build system. Go has explicit support for pkg-config in the [CGO package](https://pkg.go.dev/cmd/cgo#hdr-Using_cgo_with_the_go_command) to obtain the needed flags for compiling and linking of applications to other packages. Would this be something one could add?
I made a quick test with a minimal template for the pkg-config configuration file which worked in my setup. I have attached the patch. It creates a `openvds.pc` file from the `openvds.pc.in` template file stored in `CMake` via a `configure_file` step in the CMake build process. The configuration file is installed into `TARGET_DIR/lib/pkgconfig/`. If the patch would be already acceptable or only small extra work, I would submit it as merge request.
[0001-Add-pkg-config-configuration-file-generation.patch](/uploads/612c2ede69944f24f2e8cb4ec83d8927/0001-Add-pkg-config-configuration-file-generation.patch)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/149unable to install openvds on amazon linux2022-10-21T12:32:18Zsangamesh hooliunable to install openvds on amazon linuxERROR: Could not find a version that satisfies the requirement openvds (from versions: none)
ERROR: No matching distribution found for openvds
I have tried with all the options mentioned in https://packaging.python.org/en/latest/tutoria...ERROR: Could not find a version that satisfies the requirement openvds (from versions: none)
ERROR: No matching distribution found for openvds
I have tried with all the options mentioned in https://packaging.python.org/en/latest/tutorials/installing-packages/#installing-from-pypihttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/148Open local vds file with 260+ symbols in path2022-10-17T09:54:36ZArtem ВOpen local vds file with 260+ symbols in pathProblem in opening vds file with 260+ symbols path.
For example:
`D:\Documents\Documents\1234567890 168e79156-f70c-41a9-9b84-89f5fb9402e9 8e79156-f70c-41a9-9b84-89f5fb9402e9 8e79156-f70c-41a9-9b84-89f5fb9402e9 8e79156-f70c-41a9-9b84-89f...Problem in opening vds file with 260+ symbols path.
For example:
`D:\Documents\Documents\1234567890 168e79156-f70c-41a9-9b84-89f5fb9402e9 8e79156-f70c-41a9-9b84-89f5fb9402e9 8e79156-f70c-41a9-9b84-89f5fb9402e9 8e79156-f70c-41a9-9b84-89f5fb9402e9 8e79156-\Test_Lateral.StarSteer.data\sc\f1c720af-8e57-4137-874b-11f96e47bda7.vds`
can't be opened. Error:
```
{code=3 string="File::Open: The system cannot find the path specified.\r\n" }
```
YES, 260 Character Path Limit already fixed on my PC :smile:
```
bool File::Open(const std::string& filename, bool isCreate, bool isDestroyExisting, bool isWriteAccess, Error& error)
std::wstring native_name;
s2ws(_cFileName, native_name);
_pxPlatformHandleRead = CreateFileW(
native_name.c_str(),
GENERIC_READ | (isWriteAccess ? GENERIC_WRITE : 0),
FILE_SHARE_READ | FILE_SHARE_WRITE,
NULL,
dwCreationDisposition,
FILE_ATTRIBUTE_NORMAL | FILE_FLAG_RANDOM_ACCESS | FILE_FLAG_OVERLAPPED,
NULL);
}
```
generates `INVALID_HANDLE_VALUE`.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/146SEGYImport includes sas-token in VolumeDataLayout2022-10-17T09:58:21ZErlend HårstadSEGYImport includes sas-token in VolumeDataLayoutHi,
If invoking SEGYImport (v2.3.3) with a blob sas url, the sas is included as part of the metadata in VolumeDataLayout. E.g:
```
SEGYImport --url "azureSAS://myaccount.blob.core.windows.net/container" --url-connection "Suffix=?$SAS"...Hi,
If invoking SEGYImport (v2.3.3) with a blob sas url, the sas is included as part of the metadata in VolumeDataLayout. E.g:
```
SEGYImport --url "azureSAS://myaccount.blob.core.windows.net/container" --url-connection "Suffix=?$SAS" --compression-method=None "https://myaccount.blob.core.windows.net/container/path/mysegy.segy?sp=r&st=2022-09-30T09%3A32%3A03Z&se=2022-09-30T17%3A32%3A03Z&spr=https&sv=2021-06-08&sr=b&sig=dy%2FVXSzS15yjFgMuk0rSYzh2eIXRdjHf0MyaxxLRgI%3D"
```
produces the following entries in the VolumeDataLayout
```
{
"category" : "ImportInformation",
"name" : "DisplayName",
"type" : "String",
"value" : "mysegy.segy?sp=r&st=2022-09-30T09%3A32%3A03Z&se=2022-09-30T17%3A32%3A03Z&spr=https&sv=2021-06-08&sr=b&sig=dy%2FVXSzS15yjFgMuk0rSYzh2eIXRdjHf0MyaxxLRgI%3D"
},
{
"category" : "ImportInformation",
"name" : "InputFileName",
"type" : "String",
"value" : "mysegy.segy?sp=r&st=2022-09-30T09%3A32%3A03Z&se=2022-09-30T17%3A32%3A03Z&spr=https&sv=2021-06-08&sr=b&sig=dy%2FVXSzS15yjFgMuk0rSYzh2eIXRdjHf0MyaxxLRgI%3D"
},
```https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/145Wrong error message in OpenVDSInterfaceImpl::Close2022-09-26T14:42:45ZAnatoly YanchevskyWrong error message in OpenVDSInterfaceImpl::CloseIn function `OpenVDSInterfaceImpl::Close`:
`throw InvalidOperation(fmt::format("Flush in Close failed: {}", error.string).c_str());`
need to change on:
`throw InvalidOperation(fmt::format("Flush in Close failed: {}", flashError.string...In function `OpenVDSInterfaceImpl::Close`:
`throw InvalidOperation(fmt::format("Flush in Close failed: {}", error.string).c_str());`
need to change on:
`throw InvalidOperation(fmt::format("Flush in Close failed: {}", flashError.string).c_str());`https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/144Build OpenVDS+ with the latest msvc 20222022-09-26T09:44:53ZArtem ВBuild OpenVDS+ with the latest msvc 2022Can you provide information about msvc complier version of openvds+ (2.4.3) build?
`msvc_140` - it is actual msvc toolset, isn't it?
Do you have any plans to migrate to newly msvc (2022) and update dlls?Can you provide information about msvc complier version of openvds+ (2.4.3) build?
`msvc_140` - it is actual msvc toolset, isn't it?
Do you have any plans to migrate to newly msvc (2022) and update dlls?