osdu-airflow-lib merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests2024-02-21T14:08:52Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/234improve error message incase of jsondecode error2024-02-21T14:08:52ZVidyaDharani Lokamimprove error message incase of jsondecode error- Currently, when attempting to ingest data by uploading a file, the DAG fails with a `JSONDecodeError` if the file format is anything other than binary. To enhance the user experience, this MR improves error handling by providing a more...- Currently, when attempting to ingest data by uploading a file, the DAG fails with a `JSONDecodeError` if the file format is anything other than binary. To enhance the user experience, this MR improves error handling by providing a more informative error message in case of `JSONDecodeError`.M23 - Release 0.26VidyaDharani LokamVidyaDharani Lokamhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/222Merge branch 'fix_datasets_id_in_wpc' into 'master'2024-01-09T17:03:13ZAshish SaxenaMerge branch 'fix_datasets_id_in_wpc' into 'master'correctly replacing ACL in data records to provide for Ingestion
See merge request osdu/platform/data-flow/ingestion/osdu-airflow-lib!221
(cherry picked from commit f8d783728699ccff04091a5af214533b5c9e24a6)
4f2773c9 replacing acl base...correctly replacing ACL in data records to provide for Ingestion
See merge request osdu/platform/data-flow/ingestion/osdu-airflow-lib!221
(cherry picked from commit f8d783728699ccff04091a5af214533b5c9e24a6)
4f2773c9 replacing acl based on dummy parent enabled value
b8aec469 replacing acl based on dummy parent enabled value
e851e6ad replacing acl based on dummy parent enabled value
883cb175 replacing acl based on dummy parent enabled value
f813e6f0 debug UT output
6eb359b9 debug UT
98e76b33 debug uT
b16df9db fixed 'is_dummy_parent_mapping_enabled' var
b47e44ee fixed is_dummy_parent_mapping_enabled variable
abc8829b debugging UTM22 - Release 0.25Chad LeongChad Leonghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/221correctly replacing ACL in data records to provide for Ingestion2024-01-09T16:35:49ZPriyanka Bhongadecorrectly replacing ACL in data records to provide for IngestionM22 - Release 0.25Priyanka BhongadePriyanka Bhongadehttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/220Cherry-pick 'Update wpc_dataset_processing.py , adding correct Dataset variab...2024-01-09T12:28:21ZChad LeongCherry-pick 'Update wpc_dataset_processing.py , adding correct Dataset variable from...' into release/0.25**Original MR**: !219
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !219
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/pipelines/new?ref=cherry-pick-for-219)M22 - Release 0.25David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/219Update wpc_dataset_processing.py , adding correct Dataset variable from...2024-01-09T12:16:01ZPriyanka BhongadeUpdate wpc_dataset_processing.py , adding correct Dataset variable from...Update wpc_dataset_processing.py , adding correct Dataset variable from Constant file while changing File.Generic to CS.GenericUpdate wpc_dataset_processing.py , adding correct Dataset variable from Constant file while changing File.Generic to CS.GenericM22 - Release 0.25Priyanka BhongadePriyanka Bhongadehttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/207Cherry-pick 'Update requirements-dev.txt with package requests-aws4auth' into...2023-12-05T10:42:47ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Update requirements-dev.txt with package requests-aws4auth' into release/0.24**Original MR**: !203
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !203
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/pipelines/new?ref=cherry-pick-for-203)M21 - Release 0.24David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/203Update requirements-dev.txt with package requests-aws4auth2023-12-04T16:53:29ZPriyanka BhongadeUpdate requirements-dev.txt with package requests-aws4authhttps://gitlab.opengroup.org/osdu/subcommittees/ea/projects/extern-data/home/-/issues/310https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/extern-data/home/-/issues/310M21 - Release 0.24Priyanka BhongadePriyanka Bhongadehttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/198GONRG-8835: Error handling2024-03-25T12:13:27ZYan Sushchynski (EPAM)GONRG-8835: Error handlingM23 - Release 0.26https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/192Removed usage of vds url and persistent location from execution context.2023-11-06T16:36:21ZDeepa KumariRemoved usage of vds url and persistent location from execution context.Fixes Issue https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-vds-conversion/-/issues/17
Depends upon: https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/68
Sho...Fixes Issue https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-vds-conversion/-/issues/17
Depends upon: https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/68
Should be used with the changes in above MR.M22 - Release 0.25Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/153Cherry-pick: Merge branch 'feature-m20-wpc-handle-all-dataset-priyanka' into ...2023-09-11T08:57:11ZAshish SaxenaCherry-pick: Merge branch 'feature-m20-wpc-handle-all-dataset-priyanka' into 'release/0.23'Feature m20 wpc handle all dataset
See merge request osdu/platform/data-flow/ingestion/osdu-airflow-lib!152
(cherry picked from commit baac6a5602b65b7a488bc3aa5ce5c759b30b2477)
9e0a1f7d Update constants.py , Added DATASETS constant v...Feature m20 wpc handle all dataset
See merge request osdu/platform/data-flow/ingestion/osdu-airflow-lib!152
(cherry picked from commit baac6a5602b65b7a488bc3aa5ce5c759b30b2477)
9e0a1f7d Update constants.py , Added DATASETS constant variable
8f14052d Update wpc_dataset_processing.py, checking for Constant.DATSETSM20 - Release 0.23Chad LeongChad Leonghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/152Feature m20 wpc handle all dataset priyanka2023-09-11T08:57:34ZPriyanka BhongadeFeature m20 wpc handle all dataset priyankaM20 - Release 0.23Priyanka BhongadePriyanka Bhongadehttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/151Cherry-pick: Merge branch 'm20-parent-data-mapping' into 'release/0.23'2023-09-04T18:48:44ZChad LeongCherry-pick: Merge branch 'm20-parent-data-mapping' into 'release/0.23'M20 parent data mapping
See merge request osdu/platform/data-flow/ingestion/osdu-airflow-lib!148
(cherry picked from commit f3f7b2ea3fec638de8587bb52e7e687dfd395792)
7cffceec ParentDataMapping to self
8da27ae7 Merge branch 'eds-M20-fe...M20 parent data mapping
See merge request osdu/platform/data-flow/ingestion/osdu-airflow-lib!148
(cherry picked from commit f3f7b2ea3fec638de8587bb52e7e687dfd395792)
7cffceec ParentDataMapping to self
8da27ae7 Merge branch 'eds-M20-feature-dev' into 'm20-parent-data-mapping'
a2629ed4 To reset self.matching_attributes_per_record
24e79f73 Merge branch 'm20-parent-data-mapping' of...M20 - Release 0.23https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/150Merge branch 'eds-M20-feature-dev' into 'release/0.23'2023-09-04T18:47:31ZChad LeongMerge branch 'eds-M20-feature-dev' into 'release/0.23'abstraction for data processing
See merge request osdu/platform/data-flow/ingestion/osdu-airflow-lib!145
(cherry picked from commit 6126346a872a120b535f9383e50ba573ad1d4b39)
2e8f4f64 Merge branch 'eds-M20-feature-dev' into 'eds_natura...abstraction for data processing
See merge request osdu/platform/data-flow/ingestion/osdu-airflow-lib!145
(cherry picked from commit 6126346a872a120b535f9383e50ba573ad1d4b39)
2e8f4f64 Merge branch 'eds-M20-feature-dev' into 'eds_naturalization'
96be17e9 Merge branch 'eds-M20-feature-dev' into 'eds_naturalization'
9888389b Merge branch 'eds-M20-feature-dev' into 'eds_naturalization'
e5ede517 added refactored email automation
0d41aaad modified import statement for email automation
d56f8d36 cleaned code
fcdd2348 clear code
8a510eba Merge branch 'eds_naturalization' into 'eds-M20-feature-dev'
f79e1430 Update clean_records.py , data processing Abstraction applied
9c58fd4f Update src_dags_fetch_and_ingest.py , changed function name
26f01cfd Merge branch 'master' into 'eds-M20-feature-dev'M20 - Release 0.23https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/149Cherry-pick 'M20 parent data mapping' into release/0.232023-09-04T06:37:14ZSrinivasan NarayananCherry-pick 'M20 parent data mapping' into release/0.23**Original MR**: !142
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !142
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/pipelines/new?ref=cherry-pick-for-142)M20 - Release 0.23David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/148M20 parent data mapping2023-09-04T18:46:59ZJeyakumar DevarajuluM20 parent data mappingFixed an issue on resetting a variable while framing a queryFixed an issue on resetting a variable while framing a queryM20 - Release 0.23Jeyakumar DevarajuluJeyakumar Devarajuluhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/145abstraction for data processing2023-09-04T18:47:45ZPriyanka Bhongadeabstraction for data processingM20 - Release 0.23Priyanka BhongadePriyanka Bhongadehttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/142M20 parent data mapping2023-09-04T06:36:57ZJeyakumar DevarajuluM20 parent data mapping* Code changes to handle dummy Parent data mapping, in case the parent data not found at target system
* Code changes to handled multiple parentdataruleset conditions
* Code changes to handle more than master data like Wellbore and Organ...* Code changes to handle dummy Parent data mapping, in case the parent data not found at target system
* Code changes to handled multiple parentdataruleset conditions
* Code changes to handle more than master data like Wellbore and OrganisationM20 - Release 0.23Jeyakumar DevarajuluJeyakumar Devarajuluhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/90Feature m20 solid principle priyanka2023-08-25T07:48:39ZJeyakumar DevarajuluFeature m20 solid principle priyankaTo get the latest changesTo get the latest changesM20 - Release 0.23Jeyakumar DevarajuluJeyakumar Devarajuluhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/74Cherry-pick 'Fixed a bug to enable well record testing for parent data mappin...2023-07-14T11:50:54ZChad LeongCherry-pick 'Fixed a bug to enable well record testing for parent data mapping' into release/0.22**Original MR**: !72
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporatin...**Original MR**: !72
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/pipelines/new?ref=cherry-pick-for-72)M19 - Release 0.22David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/72Fixed a bug to enable well record testing for parent data mapping2023-07-26T05:10:26ZJeyakumar DevarajuluFixed a bug to enable well record testing for parent data mappingFixed a bug that will block the parent data mapping for well record.Fixed a bug that will block the parent data mapping for well record.M19 - Release 0.22Jeyakumar DevarajuluJeyakumar Devarajulu