Open VDS merge requestshttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/merge_requests2022-08-03T08:17:14Zhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/merge_requests/647GONRG-5236: Open VDS release image2022-08-03T08:17:14ZYauheni Rykhter (EPAM)GONRG-5236: Open VDS release imageYauheni Rykhter (EPAM)Yauheni Rykhter (EPAM)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/merge_requests/771[GONRG-6460] Add openvds GCR containerize pipeline2023-03-10T23:21:10ZDanylo Vanin (EPAM)[GONRG-6460] Add openvds GCR containerize pipelineDanylo Vanin (EPAM)Danylo Vanin (EPAM)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/merge_requests/637Guard using generator expressions in ExternalProject_add2022-06-17T09:48:28ZJørgen Lindjorgen.lind@3lc.aiGuard using generator expressions in ExternalProject_addJørgen Lindjorgen.lind@3lc.aiJørgen Lindjorgen.lind@3lc.aihttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/merge_requests/636Guard using generator expressions in ExternalProject_add2022-06-17T09:13:51ZJørgen Lindjorgen.lind@3lc.aiGuard using generator expressions in ExternalProject_addJørgen Lindjorgen.lind@3lc.aiJørgen Lindjorgen.lind@3lc.aihttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/merge_requests/42Handle non-standard SEG-Y metadata created by the Headwave SEG-Y importer.2020-02-13T10:13:08ZMorten OfstadHandle non-standard SEG-Y metadata created by the Headwave SEG-Y importer.The Bluware Headwave product predates the VDS specification and we have a lot of data that has non-standard metadata that stores converted SEG-Y text (converted to ASCII) and binary (converted to little-endian) headers. In order to make ...The Bluware Headwave product predates the VDS specification and we have a lot of data that has non-standard metadata that stores converted SEG-Y text (converted to ASCII) and binary (converted to little-endian) headers. In order to make testing easier this change adds support for these non-standard metadata to the SEGYExport tool.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/merge_requests/644Implement automatic LOD generation2022-07-21T13:14:50ZMorten OfstadImplement automatic LOD generation* Added DownSampleAndCopyRegion() function that is going to be used for LOD generation.
* Fix spelling of sse2neon target in CMakeLists, add sse2neon to .gitignore.
* Allow remapping between different LODs and call DownSampleAndCopyRegio...* Added DownSampleAndCopyRegion() function that is going to be used for LOD generation.
* Fix spelling of sse2neon target in CMakeLists, add sse2neon to .gitignore.
* Allow remapping between different LODs and call DownSampleAndCopyRegion instead of DispatchBlockCopy from VolumeDataRequestProcessor::RequestRemap() in case of LOD generation.
* Fix size[1] being passed instead of size[2] and wrong determination of lastValid[XYZ].
* Added GenerateLOD test to check that LOD generation works as expected.
* Added code to create and keep track of parent VolumeDataPageAccessors/VolumeDataPages to automatically update LODs (VolumeDataPage::WriteIntoLOD doesn't do anything yet).
* Move BlockCopy/FixupBorder/DownSampleAndCopyRegion functions from VolumeDataRequestProcessor.cpp to DataBlock.cpp/DataBlock.h.
* Use only the global VolumeDataFormat/VolumeDataComponents enums in DataBlock to remove dependency on VolumeDataChannelDescriptor.
* Implemented WriteIntoLOD method for VolumeDataPage.
* Do not generate missing data errors from sparse VDSs.
* Add separate test for automatically creating LODs.
* Make parent page dirty and update hash in WriteIntoLOD().
* Update SEGYImport with options to control LOD generation.
* Use int64_t for chunk indices.Morten OfstadMorten Ofstadhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/merge_requests/12Implement clone api2020-01-03T14:52:38ZJørgen Lindjorgen.lind@3lc.aiImplement clone apihttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/merge_requests/83Implement Curl based IOManager2020-04-07T22:21:47ZJørgen Lindjorgen.lind@3lc.aiImplement Curl based IOManagerAnd add Azure Presigned (Shared access signature) IOManager as an
implementaion example for how to implement this for other cloud vendorsAnd add Azure Presigned (Shared access signature) IOManager as an
implementaion example for how to implement this for other cloud vendorshttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/merge_requests/339Implemented accumulation of uncompressed and adaptive sizes2021-03-11T10:23:36ZMorten OfstadImplemented accumulation of uncompressed and adaptive sizesImplemented accumulation of uncompressed and adaptive sizes when writing data.Implemented accumulation of uncompressed and adaptive sizes when writing data.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/merge_requests/620Implemented new GetChunkVolumeDataHash method for VolumeDataPageAccessor2022-06-01T16:11:25ZMorten OfstadImplemented new GetChunkVolumeDataHash method for VolumeDataPageAccessorImplemented new GetChunkVolumeDataHash for VolumeDataPageAccessor which will be used to implement resumable uploads. Added utility functions to check if a hash is defined/novalue/constant.Implemented new GetChunkVolumeDataHash for VolumeDataPageAccessor which will be used to implement resumable uploads. Added utility functions to check if a hash is defined/novalue/constant.Morten OfstadMorten Ofstadhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/merge_requests/781Implement Overwrite semantics in IOManagerDmsProxy2023-03-14T09:15:34ZJørgen Lindjorgen.lind@3lc.aiImplement Overwrite semantics in IOManagerDmsProxyThis replicates the default OpenVDS behavior. We will change the default
behavior in the future to not be overwrite as default, but atleast now
they all work the same. This is also the behavior of the existing but
deprecated sdapi based ...This replicates the default OpenVDS behavior. We will change the default
behavior in the future to not be overwrite as default, but atleast now
they all work the same. This is also the behavior of the existing but
deprecated sdapi based dms iomanager.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/merge_requests/545Implement remapping between volume data layers2022-02-16T09:53:06ZMorten OfstadImplement remapping between volume data layers* Implemented lazy determination of remap source layer.
* Fix wrong error handling if zlib uncompress fails in DeserializeVolumeData.
* Converted an error case in CopyDataBlockIntoLinearBuffer into an assert since it cannot happen if the...* Implemented lazy determination of remap source layer.
* Fix wrong error handling if zlib uncompress fails in DeserializeVolumeData.
* Converted an error case in CopyDataBlockIntoLinearBuffer into an assert since it cannot happen if the code around it is correct.
* Implemented volume data hash tracking.
* Fix problem with dereferencing nullptr in VolumeDataLayer::GetProduceStatus and don't call this method in FindLayerToRemapFrom as it leads to problems with re-entrancy.
* Added getter for VolumeDataHash to VolumeDataPageImpl.
* Implemented VolumeDataRequestProcessor::RequestRemap().
* Throw runtime error instead of aborting the process in case of illegal voxel format in calls to GetVoxelFormatByteSize/GetElementSize.
* Added RequestVolumeSubsetWithDifferentFormatsAndDimensionGroups to data access tests.
* Only serialize layer status for layers that have a metadatamanager, this avoids adding information for remapped layers that are not actually stored on the remote.
* Temporarily disable remapping from lower LOD until automatic LOD creation has been implemented.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/merge_requests/775Implement the possibility of creating cloud VDSs without metadata pages2023-03-09T13:23:07ZMorten OfstadImplement the possibility of creating cloud VDSs without metadata pages* Refactor fill3DVDSWithNoise/fill3DVDSWithBitNoise to use a common fillVolumeDataPages utility function so tests can easily fill VDSs with any kind of data and also customize the parameters for the VolumeDataPageAccessor.
* Implement th...* Refactor fill3DVDSWithNoise/fill3DVDSWithBitNoise to use a common fillVolumeDataPages utility function so tests can easily fill VDSs with any kind of data and also customize the parameters for the VolumeDataPageAccessor.
* Implement the possibility of creating cloud VDSs without metadata pages by passing a chunkMetadataPageSize of 0 when creating the VolumeDataPageAccessor (not possible with VDS files).https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/merge_requests/576Implement WriteLock for VolumeDataLayer to avoid errors with multiple VolumeD...2022-03-10T10:33:30ZMorten OfstadImplement WriteLock for VolumeDataLayer to avoid errors with multiple VolumeDataPageAccessors* Add option to WriteLock VolumeDataLayers in order to check that we don't create multiple VolumeDataPageAccessors for layers while writing data.
* Add code to VolumeDataPageAccessor to acquire the VolumeDataLayer WriteLock in CreatePage...* Add option to WriteLock VolumeDataLayers in order to check that we don't create multiple VolumeDataPageAccessors for layers while writing data.
* Add code to VolumeDataPageAccessor to acquire the VolumeDataLayer WriteLock in CreatePage()/GetWritableBuffer() and release it in Commit().
* Add check for write-locked layers in HueVolumeDataPageAccessor_c::ReadPage and throw exception if reading from another accessor than the one that has the write lock.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/merge_requests/65Improve colormap output in Jupyter Notebook example2020-03-11T08:47:03ZJørgen Lindjorgen.lind@3lc.aiImprove colormap output in Jupyter Notebook examplehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/merge_requests/55Improve detection of python2020-02-27T10:28:27ZJørgen Lindjorgen.lind@3lc.aiImprove detection of pythonhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/merge_requests/871Improved parameter validation2023-11-30T14:05:58ZMorten OfstadImproved parameter validationImproved parameter validation of dimension parameter for methods on the VolumeDataLayout so it is consistent with commercial VDS implementation.Improved parameter validation of dimension parameter for methods on the VolumeDataLayout so it is consistent with commercial VDS implementation.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/merge_requests/348Improved Python create VDS test2021-03-12T10:30:04ZMorten OfstadImproved Python create VDS testFixed the array selector of the target buffer. It actually worked before, but only by accident because Python limits the selector to the actual shape of the array and the max[] was always larger than the shape of the array. Since we alwa...Fixed the array selector of the target buffer. It actually worked before, but only by accident because Python limits the selector to the actual shape of the array and the max[] was always larger than the shape of the array. Since we always want to select the whole target buffer array we just use the universal selector for the whole array instead.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/merge_requests/793Improve error handling2023-03-29T14:16:19ZMorten OfstadImprove error handling* Make VolumeDataRequest::GetErrorCode()/GetErrorMessage() wait for the request to complete (if it hasn't done so already) so they can return sensible values.
* Turn the 'inconsistent metadata' case back into an error. It looks like it w...* Make VolumeDataRequest::GetErrorCode()/GetErrorMessage() wait for the request to complete (if it hasn't done so already) so they can return sensible values.
* Turn the 'inconsistent metadata' case back into an error. It looks like it was changed by accident when the logging interface was added.
* Update tests to check that the expected errors are logged and can be picked up through the VolumeDataRequest object.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/merge_requests/300Improve error handling in SliceDump2021-01-27T09:35:41ZJørgen Lindjorgen.lind@3lc.aiImprove error handling in SliceDump