Open ZGY merge requestshttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/merge_requests2024-03-19T17:06:46Zhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/merge_requests/151More end-user documentation.2024-03-19T17:06:46ZPaal KvammeMore end-user documentation.Explain in the top level readme file how to obtain a binary distribution.
Also mention in the readme the existing C++ example code, and a new Python example.Explain in the top level readme file how to obtain a binary distribution.
Also mention in the readme the existing C++ example code, and a new Python example.M23 - Release 0.26Paal KvammePaal Kvammehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/merge_requests/150Add --legaltag to the command line options for zgycopyc2024-03-19T17:06:47ZPaal KvammeAdd --legaltag to the command line options for zgycopycAdd --legaltag to the command line options for zgycopyc.\
The code is backported from the SLB-Internal ZgyTool.\
To copy the output tag from the input, use '--legaltag=\*'.\
Copying involves an extra open/close of the input.\
That might ...Add --legaltag to the command line options for zgycopyc.\
The code is backported from the SLB-Internal ZgyTool.\
To copy the output tag from the input, use '--legaltag=\*'.\
Copying involves an extra open/close of the input.\
That might be fixed someday.\
The default legaltag is '', i.e. use the project default.\
This differs from ZgyTool which defaults to '\*'.M23 - Release 0.26Paal KvammePaal Kvammehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/merge_requests/149Sync build rules from the SLB-internal repository2024-03-19T17:06:48ZPaal KvammeSync build rules from the SLB-internal repositorySync build rules from the SLB-internal repository.
Some of the updates, such as fixing compiler warnings, also apply to the open source repository. The rest is being merged just to keep the repositories as similar as possible.Sync build rules from the SLB-internal repository.
Some of the updates, such as fixing compiler warnings, also apply to the open source repository. The rest is being merged just to keep the repositories as similar as possible.M23 - Release 0.26Paal KvammePaal Kvammehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/merge_requests/148SAuth support in all builds2024-03-19T17:06:50ZPaal KvammeSAuth support in all buildsAdd support for an authentication framework used inside SLB. It will probably not be useful for others. Basically, the change just implements OAuth-like rules for talking to the SLB license servers.Add support for an authentication framework used inside SLB. It will probably not be useful for others. Basically, the change just implements OAuth-like rules for talking to the SLB license servers.M23 - Release 0.26Paal KvammePaal Kvammehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/merge_requests/147Add rocky8 and omega pipelines2024-01-22T17:14:20ZPaal KvammeAdd rocky8 and omega pipelinesAdd support for rocky8 and omega in the OSDU build rules.
rocky8 binaries can be used for distros based on Red Hat 8 or CentOS 8.
omega is a special version of rocky8 where the c++ ABI uses the old definitions (pre-c++11) for std::stri...Add support for rocky8 and omega in the OSDU build rules.
rocky8 binaries can be used for distros based on Red Hat 8 or CentOS 8.
omega is a special version of rocky8 where the c++ ABI uses the old definitions (pre-c++11) for std::string. This allows linking with legacy libraries that cannot be rebuilt. Google \_GLIBCXX_USE_CXX11_ABI for details.M23 - Release 0.26Paal KvammePaal Kvammehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/merge_requests/146Add code to read legaltag.2024-01-22T17:14:22ZPaal KvammeAdd code to read legaltag.Add code to read the legaltag from a file stored in sdms. The file does not need to be open, and does not need to be a ZGY file. To be used by zgytool to propagate the legaltag when copying. Might also be used directly from application c...Add code to read the legaltag from a file stored in sdms. The file does not need to be open, and does not need to be a ZGY file. To be used by zgytool to propagate the legaltag when copying. Might also be used directly from application code, although that has not been requested yet.
It is technically possible to instead read the tag from an already open file in order to avoid the time spent to open and close it. However, this is more work to implement and/or will break encapsulation. And it would need to be implemented separately for ZGY and VDS.M23 - Release 0.26Paal KvammePaal Kvammehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/merge_requests/145Sync with the SLB-internal OpenZGY repository2024-01-22T17:14:24ZPaal KvammeSync with the SLB-internal OpenZGY repositoryPull changes made to the SLB internal repository that are not really relevant for the OSDU build. But it is practical to keep things in sync.
Internal pull requests for Azure DevOps: 549297 550262 550295 and for the expected locaton of ...Pull changes made to the SLB internal repository that are not really relevant for the OSDU build. But it is practical to keep things in sync.
Internal pull requests for Azure DevOps: 549297 550262 550295 and for the expected locaton of test data: 555768M23 - Release 0.26Paal KvammePaal Kvammehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/merge_requests/144Add missing header that causes trouble for clang 16.2024-01-04T11:30:13ZPaal KvammeAdd missing header that causes trouble for clang 16.Add missing header that causes trouble for clang 16.
Reported as Issue #31
I have not observed the error myself. I have a Docker pipeline for testing clang 14 in Ubuntu Jammy, but I can't find clang 16 in the Docker repositories. Addi...Add missing header that causes trouble for clang 16.
Reported as Issue #31
I have not observed the error myself. I have a Docker pipeline for testing clang 14 in Ubuntu Jammy, but I can't find clang 16 in the Docker repositories. Adding a "clang 16" to our internal build rules would be tedious. So, there might be other problems.
The fix is completely safe, though. For reasons of general paranoia I am still going to wait for out internal build pipelines to complete before I complete this request.M23 - Release 0.26Paal KvammePaal Kvammehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/merge_requests/143Use SDManager::create2023-12-07T16:28:23ZPaal KvammeUse SDManager::createOpenZGY is changed to use SDManager::create(serialized) instead of calling the constructor and then the appropriate setAuthProviderXXX(). This allows using credential types that are not visible in the API. And potentially not even presen...OpenZGY is changed to use SDManager::create(serialized) instead of calling the constructor and then the appropriate setAuthProviderXXX(). This allows using credential types that are not visible in the API. And potentially not even present when libsdapi.so was built.
Currently this change is not much help, since the overload of SDManager::create needed to get this to work is one of those methods that are deemed SLB private and needs to be patched into SDManager. So this pull request essentially switches from using one set of SLB private methods to another.
If the open source version of SDAPI is changed to support SAuth, even if it doesn't show up in the API, then this allows OpenZGY to use the new credential types without maintaining two separate sets of build pipelines. And, it allows applications linking with SDAPI from OSDU to use that functionality.
Note that only normal tokens and client credentials grant have been tested. Because I don't know how to create the other SAuth types. The risk of those not working is high. But any bugs in that code will not affect those types that have been tested.M22 - Release 0.25Paal KvammePaal Kvammehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/merge_requests/142Improve code coverage in unit tests2023-12-07T16:28:24ZPaal KvammeImprove code coverage in unit testsImprove code coverage by amending old and writing new unit tests.Improve code coverage by amending old and writing new unit tests.M22 - Release 0.25Paal KvammePaal Kvammehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/merge_requests/141Devops work for the OSDU pipelines2023-11-28T12:50:21ZPaal KvammeDevops work for the OSDU pipelinesDevops work for the OSDU pipelines.
- Refactor the GitLab pipeline to fix bad assumptions.
- Don't hard code versions of PyPI packages.Devops work for the OSDU pipelines.
- Refactor the GitLab pipeline to fix bad assumptions.
- Don't hard code versions of PyPI packages.M22 - Release 0.25Paal KvammePaal Kvammehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/merge_requests/140Low hanging fruit2023-11-28T12:50:22ZPaal KvammeLow hanging fruitAdd some bells and whistles, and do some cleanup.
* Add support for -O and -S options in zgydumpc.
* Minor speedup when not generating low resolution data.
* Sync with internal SLB devops. In OSDU this is just for information.
* Other c...Add some bells and whistles, and do some cleanup.
* Add support for -O and -S options in zgydumpc.
* Minor speedup when not generating low resolution data.
* Sync with internal SLB devops. In OSDU this is just for information.
* Other cleanup.M22 - Release 0.25Paal KvammePaal Kvammehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/merge_requests/139Merge internal PR 525649, 534872, 537022: Heuristic for lowres data present2023-11-28T12:50:24ZPaal KvammeMerge internal PR 525649, 534872, 537022: Heuristic for lowres data presentFix the heuristic that decides when low resolution data is present.
One of the new features in OpenZGY is the ability to create a ZGY file with no resolution data. Application code will see nlods() returning 1.
The change was done with...Fix the heuristic that decides when low resolution data is present.
One of the new features in OpenZGY is the ability to create a ZGY file with no resolution data. Application code will see nlods() returning 1.
The change was done without breaking the layout of ZGY files. Instead of storing some usable_nlods, the reader uses a heuristic to decide whether low resolution bricks are present. That heuristic turned out to be a lot more complex than expected. Especially when factoring in corner cases such as a constant-value survey (doesn't need finalizing for the lowres to be valid) or single-brick surveys (no lowres needed, so finalize does nothing).M22 - Release 0.25Paal KvammePaal Kvammehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/merge_requests/138Multiple retries when running pip.2023-11-28T12:50:27ZPaal KvammeMultiple retries when running pip.Retry a failing "pip" command with successively longer timeouts. This reduces the risk but doesn't guarantee that gitlab builds won't time out. Not all the "pip" invocations are replaced with the retry wrapper. Add more if needed. Note t...Retry a failing "pip" command with successively longer timeouts. This reduces the risk but doesn't guarantee that gitlab builds won't time out. Not all the "pip" invocations are replaced with the retry wrapper. Add more if needed. Note that all failures are retried, not just timeouts. That is a cosmetic issue only.M22 - Release 0.25Paal KvammePaal Kvammehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/merge_requests/137Major update to use single pass write, brick api, and more2023-11-28T12:50:28ZPaal KvammeMajor update to use single pass write, brick api, and moreThis is a major rewrite of the OpenZGY bulk layer. The goal is to speed up writes.
Low resolution bricks will now be written interleaved with full resolution data instead of written in a separate pass. The pull request includes other pe...This is a major rewrite of the OpenZGY bulk layer. The goal is to speed up writes.
Low resolution bricks will now be written interleaved with full resolution data instead of written in a separate pass. The pull request includes other performance improvements as well.
The pull request has significant risk of bugs due to the amount of changes. Roughly 170 commits adding 14% and removing 5% of the initial 65,000 lines of code.
A more detailed explanation of this pull request can be found in [doc/single-pass-write.md](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/blob/master/doc/single-pass-write.md).M22 - Release 0.25Paal KvammePaal Kvammehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/merge_requests/136Dynamic histogram, GIL, and more.2023-08-18T10:32:07ZPaal KvammeDynamic histogram, GIL, and more.Commit changes from the internal SLB repository.
### Python GIL unlock
Merged from PR 477598
Opt-in feature where the Python GIL is released during long running I/O. This allows for multi-threaded Python scripts. The new feature is no...Commit changes from the internal SLB repository.
### Python GIL unlock
Merged from PR 477598
Opt-in feature where the Python GIL is released during long running I/O. This allows for multi-threaded Python scripts. The new feature is not default because the documentation is a bit vague about what the code can do when the GIL is not held. The feature is also not useful if the Python script configures OpenZGY to do multi-threading itself.
### Dynamically extendable histogram
Merged from PR 483423
Implement a histogram that can collect information in a single pass. Without prior knowledge of the value range of the input samples.
The code will eventually be used inside OpenZGY. It is being exposed in the API as a convenience for applications that want to make a histogram but aren't about to write a ZGY file.
The histogram range will rarely be perfect. It might not make use of the entire range because of the heuristics used. There are also rare cases, e.g. a sample value range of 100,000 to 100,010, where the histogram will be useless. Most cases should give a reasonable result.
The histogram will be zero-centric, meaning that the sample value 0 will be at the center of a bin. Note that if the histogram is built with integral sample values that will later be scaled to float, then the application is responsible for keeping the result zero-centric.
### Testing and bugfixing
Merged from PR 483669
Fix an obscure bug with constant value bricks where the value was "1".
Add unit tests for this corner case, and also several regression tests that I want to be in place before starting to push some significant changes.
### Statistical range
Merged from PR 483652
If spikes are written to a ZGY file and later overwritten, the statistical min/max range will still include the spikes.
The PR ***removes*** code that tries to mitigate the problem by looking at the histogram. See comments in the source code for why.
Arguably the feature isn't that useful anyway. The feature might be put back later, but handled at a higher level outside the ExpandableBuilder.M19 - Release 0.22Paal KvammePaal Kvammehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/merge_requests/135Sync with changes in the private SLB repository2023-08-18T10:32:09ZPaal KvammeSync with changes in the private SLB repositorySync with changes in the private SLB repository
- Consume a newer SDAPI in internal SLB builds.
- Add more telemetry and other debug code.
- Forgotten return statements are fatal in gcc.
- Make IOContext changes less likely to break the...Sync with changes in the private SLB repository
- Consume a newer SDAPI in internal SLB builds.
- Add more telemetry and other debug code.
- Forgotten return statements are fatal in gcc.
- Make IOContext changes less likely to break the ABI.
- Add httpHeaders() to the IOContext.
### IOContext details:
The existing class IOContext and derived implements the named parameter idiom for configuring I/O.
Make IOContext changes less likely to break the ABI. Introduce a new SeismicStoreIOContextV2 that uses the pimpl pattern and that has no inlined functions. The main drawback is that a significant code bloat, and somewhat higher maintenance cost.
To avoid breaking the ABI right now, SeismicStoreIOContextV2 inherits from the unchanged SeismicStoreIOContext. So the pimpl pattern only applies to new members. On the next major release, consider moving everything into the pimpl.
Note the following minor inconvenience: Parameters in the new V2 class needs to be set before those in the old class. I could fix this, but that would lead to even more bloat.
This works:
```
auto args = SeismicStoreIOContextV2().newparam(x).oldparam(y);
```
Also the longer version:
```
SeismicStoreIOContextV2 args;
args.oldparam(x);
args.newparam(x); // either order.
```
This will not compile:
```
auto args = SeismicStoreIOContextV2().oldparam(x).newparam(y);
```
The pull request adds a single new paramater using the new class. httpHeaders() will be passed on to SDAPI when creating an SDManager(). There is also a fallback environment variable OPENZGY_HTTP_HEADERS.
Applications that needs to use the new parameter can switch to use the new class exclusively.
### Telemetry details:
Optionally record all read() and write() calls at the OpenZGY public API level.
The result is output in CSV format to the file specified in $OPENZGY_RECORD_LOGFILE. The idea is that a log file from Petrel could be used to replay the same access pattern using a stand alone tool. This could make it much easier to debug *actual* performance issues that Petrel is seeing. Note that I have not written the replay tool, only he logger.
Caveat: For multi-threaded access, even if the replay tool is multi-threaded, the replay might not be sufficiently similar to the original. It depends on how much effort we put into that new tool. And the problem being analyzed.M19 - Release 0.22Paal KvammePaal Kvammehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/merge_requests/134Draft: Test run2023-10-11T16:17:19ZSumra ZafarDraft: Test runhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/merge_requests/133Build rules for Ubuntu Jammy with OpenZGY using clang++ and c++20. Still usin...2023-08-18T10:32:11ZPaal KvammeBuild rules for Ubuntu Jammy with OpenZGY using clang++ and c++20. Still using...Fix issues and warnings seen when building with with clang and C++20. Most of the changes have been contributed by Jon Jenssen. See issue #26.
Add build rules for Ubuntu Jammy with OpenZGY built with clang. SDAPI and dependencies are st...Fix issues and warnings seen when building with with clang and C++20. Most of the changes have been contributed by Jon Jenssen. See issue #26.
Add build rules for Ubuntu Jammy with OpenZGY built with clang. SDAPI and dependencies are still built with the gnu compilers. The new configuration has been added to the build pipelines run internally in SLB. There are currently no scheduled builds in OSDU / GitLab.M17 - Release 0.20Paal KvammePaal Kvammehttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy/-/merge_requests/132Improvements to the zgycopy tool2023-08-18T10:32:13ZPaal KvammeImprovements to the zgycopy tool- Merged PR 443683: Progress bar with ETL.
- Merged PR 444764: Bells and whistles in the OpenZGY "zgycopyc" tool.
- Merged PR 445254: Workaround for SAuth being too complex.- Merged PR 443683: Progress bar with ETL.
- Merged PR 444764: Bells and whistles in the OpenZGY "zgycopyc" tool.
- Merged PR 445254: Workaround for SAuth being too complex.M17 - Release 0.20Paal KvammePaal Kvamme