osdu-ingestion-lib merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests2024-01-02T09:36:36Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/68adding persistant file name and path fetching from segy url2024-01-02T09:36:36ZDeepa Kumariadding persistant file name and path fetching from segy urlFixes issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-vds-conversion/-/issues/17Fixes issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-vds-conversion/-/issues/17M22 - Release 0.25Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/26Add rc versions2022-08-12T14:58:13ZYan Sushchynski (EPAM)Add rc versionsM14 - Release 0.17https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/59Cherry-pick 'Enable user context ingestion' into release/0.212023-05-29T09:04:30ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Enable user context ingestion' into release/0.21**Original MR**: !53
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporatin...**Original MR**: !53
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/pipelines/new?ref=cherry-pick-for-53)M18 - Release 0.21David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/58Cherry-pick 'Fix replace authority with data-partition-id' into release/0.212023-05-25T11:50:49ZChad LeongCherry-pick 'Fix replace authority with data-partition-id' into release/0.21**Original MR**: !49
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporatin...**Original MR**: !49
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/pipelines/new?ref=cherry-pick-for-49)M18 - Release 0.21David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/75Cherry-pick 'fix: typo in RoleID' into release/0.252024-01-09T10:07:39ZChad LeongCherry-pick 'fix: typo in RoleID' into release/0.25**Original MR**: !73
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporatin...**Original MR**: !73
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/pipelines/new?ref=cherry-pick-for-73)M22 - Release 0.25David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/60default UserId as None2023-07-31T09:41:20ZDeepa Kumaridefault UserId as NoneMake the user id have a default value, so user classes won't break.
Original MR: https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/52Make the user id have a default value, so user classes won't break.
Original MR: https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/52M20 - Release 0.23Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/24Draft: Enable passing of user id from Airflow Context2023-05-16T04:47:31Zharshit aggarwalDraft: Enable passing of user id from Airflow ContextRelated ADR - https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/52
In accordance with the above ADR change involves enabling of setting the userId into Airflow context so it can be used in BaseClient Class ...Related ADR - https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/52
In accordance with the above ADR change involves enabling of setting the userId into Airflow context so it can be used in BaseClient Class in python SDK
Related MR - https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/merge_requests/66M16 - Release 0.19harshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/53Enable user context ingestion2023-05-26T16:37:55ZDeepa KumariEnable user context ingestion* ADR : https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/52
In accordance with the above ADR change involves consuming the userId passed from Airflow Conf and setting the on-behalf-of header
Related MR - ...* ADR : https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/52
In accordance with the above ADR change involves consuming the userId passed from Airflow Conf and setting the on-behalf-of header
Related MR - https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/165
User Id is a part of execution context, but is not a mandatory requirementM18 - Release 0.21Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/11feat(azure): add file handing support for azure provider2023-08-18T11:25:25ZVadzim Kulybafeat(azure): add file handing support for azure provider## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Added specific header for azure provider
Details:
https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/issues/3
Depend on:
https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/merge_requests/57M10 - Release 0.13Vadzim KulybaVadzim Kulybahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/16Fix add limit to search_record_ids2022-03-15T14:05:16ZErnesto GutierrezFix add limit to search_record_idsAdd limit to query request to allow more than 10 search results returned in response.Add limit to query request to allow more than 10 search results returned in response.M10 Patch - Release 0.13 patchhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/6Fix datetime validation2022-03-21T15:29:31ZYan Sushchynski (EPAM)Fix datetime validationDelete strict-rfc package and validation. Use data-time types as in Indexer: https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/master/indexer-core/src/test/java/org/opengroup/osdu/indexer/util/parser/DateTimePar...Delete strict-rfc package and validation. Use data-time types as in Indexer: https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/master/indexer-core/src/test/java/org/opengroup/osdu/indexer/util/parser/DateTimeParserTest.java#L31
closes #2M10 - Release 0.13Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/63fixed file-source2023-08-23T10:02:45ZDeepa Kumarifixed file-sourceChanging file source info to contain vds info. We made some changes for file source infos for https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-vds-conversion/-/issues/15.
The resultant Dataset looks like this fo...Changing file source info to contain vds info. We made some changes for file source infos for https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-vds-conversion/-/issues/15.
The resultant Dataset looks like this for now, without these changes:
```
"DatasetProperties": {
"FileCollectionPath": "sd://opendes/autotest/test_vds_id_999947655129",
"FileSourceInfos": "[[{'FileSource': 'sd://opendes/autotest/ST10010ZC11_PZ_PSDM_KIRCH_FULL_T.MIG_FIN.POST_STACK.3D.JS-017536.segy', 'PreloadFilePath': '', 'Name': 'OSDU_SEGY'}]]"
}
```
Not a properly formatted json, plus the file source should be telling about the vds file and not the segy file. So, until https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-vds-conversion/-/issues/17 gets implemented, this is a temp fix. With this, now we get:
```
"FileCollectionPath": "sd://opendes/autotest/test_vds_id_999750650895",
"FileSourceInfos": [
{
"FileSource": "sd://opendes/autotest/test_vds_id_999750650895"
}
]
```M20 - Release 0.23Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/45fix: file source validator2023-03-31T20:51:14ZYan Sushchynski (EPAM)fix: file source validatorFor fixing this issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/issues/10
It is possible to validate non-file-based datasets nowFor fixing this issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/issues/10
It is possible to validate non-file-based datasets nowM17 - Release 0.20https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/29Fix not skipped WP2022-09-30T16:44:49ZYan Sushchynski (EPAM)Fix not skipped WPFix WPs not being added to `skipped_ids` if there were issues with getting them from Schema service.Fix WPs not being added to `skipped_ids` if there were issues with getting them from Schema service.M14 - Release 0.17Vadzim KulybaVadzim Kulybahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/61fix: Read Segy SD Path from combining the FileSource and FileCollectionPath f...2023-08-03T04:47:49ZDeepa Kumarifix: Read Segy SD Path from combining the FileSource and FileCollectionPath from initial record.This MR is to fix linked issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-vds-conversion/-/issues/15This MR is to fix linked issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-vds-conversion/-/issues/15M20 - Release 0.23Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/49Fix replace authority with data-partition-id2023-05-25T09:58:35ZYan Sushchynski (EPAM)Fix replace authority with data-partition-idResolves this issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/issues/9Resolves this issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/issues/9M18 - Release 0.21https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/73fix: typo in RoleID2024-01-09T10:06:33ZIvan Medeiros Monteirofix: typo in RoleIDFix typo that makes the work product component (SeismicTraceData) non-compliant.
Issue reported here : https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/issues/12Fix typo that makes the work product component (SeismicTraceData) non-compliant.
Issue reported here : https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/issues/12M22 - Release 0.25Ivan Medeiros MonteiroIvan Medeiros Monteirohttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/50fix: update Python version in tests in .gitlab-ci.yml2023-05-05T15:22:50ZYan Sushchynski (EPAM)fix: update Python version in tests in .gitlab-ci.ymlM18 - Release 0.21https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/2GONRG-3119: Python SDK Usage2022-01-07T09:03:32ZYan Sushchynski (EPAM)GONRG-3119: Python SDK UsageAdd Python SDK `clients` instead of `requests`.Add Python SDK `clients` instead of `requests`.M10 - Release 0.13Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/33GONRG-3210: Make code to be compliant with PyType checker2023-02-21T11:51:27ZYan Sushchynski (EPAM)GONRG-3210: Make code to be compliant with PyType checkerADR: https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/15
Make the code compliant with PyTypeADR: https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/15
Make the code compliant with PyTypeM16 - Release 0.19