Segy to Zgy Conversion DAG merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests2021-09-02T13:11:58Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests/33Change segy_to_zgy_ingestion_dag imports to follow the new structure2021-09-02T13:11:58ZYan Sushchynski (EPAM)Change segy_to_zgy_ingestion_dag imports to follow the new structure## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Due to https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/67 we have to change importsM8 - Release 0.11https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests/29GONRG-2973: Added Airflow 2.0 compatibility2021-08-25T12:39:40ZSiarhei Khaletski (EPAM)GONRG-2973: Added Airflow 2.0 compatibility## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x] IBM
## Updates description?
Added support of Airflow 2.0 with backward compatibility with Airflow 1.10.*.M8 - Release 0.11Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests/40GONRG-3605: Update DAG's deps version to 0.12.02021-10-27T12:56:42ZYan Sushchynski (EPAM)GONRG-3605: Update DAG's deps version to 0.12.0## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Update requirements.txt file for Airflow environment to release onesM9 - Release 0.12Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests/36GONRG-3112: Added Update Status tasks2021-09-16T07:26:58ZSiarhei Khaletski (EPAM)GONRG-3112: Added Update Status tasks## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
SEGY->ZGY v2 DAG has been updated. Update workflow status tasks have been added.M9 - Release 0.12Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests/43Draft: Resolve "Wrong `ancestry.parents` value in `FileCollectionSlbOpenZGY` ...2021-11-16T16:44:23ZIvan Medeiros MonteiroDraft: Resolve "Wrong `ancestry.parents` value in `FileCollectionSlbOpenZGY` record."## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)
Closes #16M10 - Release 0.13Ivan Medeiros MonteiroIvan Medeiros Monteirohttps://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests/42WIP: GONRG-3690: Add Segy Zgy tests2022-01-11T15:28:59ZYan Sushchynski (EPAM)WIP: GONRG-3690: Add Segy Zgy tests## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Add E2E tests for Segy->Zgy DAG.M10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests/66Delete default schedule interval2022-04-28T16:09:42ZYan Sushchynski (EPAM)Delete default schedule interval## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
The DAG is scheduled to be executed once a day without payload, which is a default behavior of Airflow if schedule_interval is not set to `None`.
To fix this unexpected behavior, we introduce this MR.M12 - Release 0.15harshit aggarwalSiarhei Khaletski (EPAM)Shrikant GargSpencer Suttonsuttonsp@amazon.comharshit aggarwalhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests/61Automating DAG Upload2023-08-18T10:48:07ZSumra ZafarAutomating DAG Upload## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M13 - Release 0.16Sumra ZafarSumra Zafarhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests/119Feature/clean up dags and unused scripts2022-09-20T11:35:29ZVadzim KulybaFeature/clean up dags and unused scripts## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
Cleaning unused dags and scripts## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
Cleaning unused dags and scriptsM14 - Release 0.17Vadzim KulybaVadzim Kulybahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests/163fix: segy_configure_remote_access to set http headers2022-11-14T20:55:11ZIvan Medeiros Monteirofix: segy_configure_remote_access to set http headers## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
fix: segy_configure_remote_access to set http headersM15 - Release 0.18Ivan Medeiros MonteiroIvan Medeiros Monteirohttps://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests/219Integrate slb-master2023-03-31T09:30:41ZErik DahlIntegrate slb-master## Type of change
- [x] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M17 - Release 0.20Erik DahlErik Dahlhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests/218First implementation of x-on-behalf-of header for storage client and SDApi2023-03-29T10:54:35ZErik DahlFirst implementation of x-on-behalf-of header for storage client and SDApi## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M17 - Release 0.20Erik DahlErik Dahlhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests/195Major performance work, adding GSM logging module and created authentication ...2023-03-29T10:58:18ZErik DahlMajor performance work, adding GSM logging module and created authentication plugin framework module.## Type of change
- [x] Bug Fix
- [x ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [X] Azure
- [X] GCP
- [X...## Type of change
- [x] Bug Fix
- [x ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [X] Azure
- [X] GCP
- [X] IBM
## Updates description?
- Change to DAG: The environment SD_READ_CACHE_MAX_PAGES now controls number of 40MB (average value) pages of read-ahead buffers to be used. Please make sure this value is set to a reasonable value, usually a value between 16 and 64 should work, but might cause the DAG to run out of memory if set too large.
- Authentication plugin framework allows for integrating into more sophisticated authentication scheme than simply providing a JWT token, e.g. support for oauth2.
- Major rework of data access when running the SEG-Y to ZGY converter from cloud based repositoryM17 - Release 0.20Erik DahlErik Dahlhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests/290Cherry-pick 'Added changes for User-Context-Ingestion' into release/0.222023-07-14T11:50:28ZChad LeongCherry-pick 'Added changes for User-Context-Ingestion' into release/0.22**Original MR**: !289
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !289
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/pipelines/new?ref=cherry-pick-for-289)M19 - Release 0.22David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests/289Added changes for User-Context-Ingestion2023-07-14T11:01:09ZDeepa KumariAdded changes for User-Context-IngestionUser context ADR: https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/52
Previously present changes of sauth have been used after kind of a rename and completing the flow from segy_to_zgy_ingestion_dag. Also ...User context ADR: https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/52
Previously present changes of sauth have been used after kind of a rename and completing the flow from segy_to_zgy_ingestion_dag. Also some additional logging has been added.
No new changes per se.M19 - Release 0.22Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests/288Integrate master with updates from slb-master2023-07-19T21:00:43ZErik DahlIntegrate master with updates from slb-master## Type of change
- [x] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x]...## Type of change
- [x] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x] IBM
## Updates description?
* Expose OpenZGY IOCONTEXT values with new environment variables: SEGYTOZGY_IOCONTXT_BUFFER_SIZE, SEGYTOZGY_IOCONTXT_WRITE_THREADS, SEGYTOZGY_IOCONTXT_CPU_THREADS, SEGYTOZGY_IOCONTXT_IO_THREADS. Default settings will acquire less memory for IOCONTEXT than it was before.
* Handling of SGY files with incorrect Trace Header value for samples per trace.
* Removed openssl dependency
* Fixed calculation of SEG-Y Index sample rate for Depth volumes
* Fixes of executables exiting
* Load context when json contains only data member
* Optimization improvement: reuse readers for SEGY session, don't close in case of no performed writes
* Small code refactoring according to the code style and static analysis toolM19 - Release 0.22Erik DahlErik Dahlhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests/287Do not use OpenZGY write with default IOCONTEXT values2023-07-13T15:24:26ZErik DahlDo not use OpenZGY write with default IOCONTEXT values## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x] IBM
## Updates description?
Use less memory hungry values for OpenZGY's iocontext setting. The old method of using default parameter would allow the OpenZGY write cache to grow up to 2GB for large files!
The parameters can now be tuned at run time. The operation now uses the following environment variables for the iocontext:
SEGYTOZGY_IOCONTXT_BUFFER_SIZE <Default value is 32 MB>
SEGYTOZGY_IOCONTXT_WRITE_THREADS <Default value is 16 threads>
SEGYTOZGY_IOCONTXT_CPU_THREADS <Less critical parameter, default value is 1>
SEGYTOZGY_IOCONTXT_IO_THREADS <Less critical parameter, default value is 1>
Using the new default values (i.e. not setting the environment variables) results in an OpenZGY cache size of (32 * 16) = 512MbM19 - Release 0.22Erik DahlErik Dahlhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests/286Handle file like section.sgy where trace header has incorrect value for sampl...2023-07-13T15:24:41ZErik DahlHandle file like section.sgy where trace header has incorrect value for samples per trace## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Allow user to specify overrides for trace header and binary header values in VectorHeaderMappingM19 - Release 0.22Erik DahlErik Dahlhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests/276fix: increase lock timeout to avoid race condition on valid waiting2023-07-13T15:05:46ZIvan Medeiros Monteirofix: increase lock timeout to avoid race condition on valid waiting## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M19 - Release 0.22Ivan Medeiros MonteiroIvan Medeiros Monteirohttps://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/merge_requests/272Fix calculation of SEG-Y Index sample rate for Depth volumes2023-07-13T14:59:49ZErik DahlFix calculation of SEG-Y Index sample rate for Depth volumes## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Fixes a calculation error for sample rate in the SEG-Y index for DEPTH volumes. (Old version was off by a factor of 1000 ... )
The fix is required because sample rate is specified in different units for Time and Depth volumes:
For Time volumes, first and last sample are specified in milliseconds, and sample rate is in microseconds (millisecond = 1/1000 second, microsecond is 1/1000,000 second)
For Depth volumes, the sample rate is usually (in SEG-Y rev 0 and rev 1) specified in 1/1000 units, e.g. if the Depth of First and last sample is specified in metres, the sample rate (e.g. 4m) will be given a value of 4000. Before the fix, the value would be divided by 1000,000, making the reader think there were 1000 x as many samples per trace as there should have been ....
The fix is to divide Depth sample rate by 1000 rather than 1000,000M19 - Release 0.22Erik DahlErik Dahl