osdu-ingestion-lib merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests2023-08-23T10:02:45Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/63fixed file-source2023-08-23T10:02:45ZDeepa Kumarifixed file-sourceChanging file source info to contain vds info. We made some changes for file source infos for https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-vds-conversion/-/issues/15.
The resultant Dataset looks like this fo...Changing file source info to contain vds info. We made some changes for file source infos for https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-vds-conversion/-/issues/15.
The resultant Dataset looks like this for now, without these changes:
```
"DatasetProperties": {
"FileCollectionPath": "sd://opendes/autotest/test_vds_id_999947655129",
"FileSourceInfos": "[[{'FileSource': 'sd://opendes/autotest/ST10010ZC11_PZ_PSDM_KIRCH_FULL_T.MIG_FIN.POST_STACK.3D.JS-017536.segy', 'PreloadFilePath': '', 'Name': 'OSDU_SEGY'}]]"
}
```
Not a properly formatted json, plus the file source should be telling about the vds file and not the segy file. So, until https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-vds-conversion/-/issues/17 gets implemented, this is a temp fix. With this, now we get:
```
"FileCollectionPath": "sd://opendes/autotest/test_vds_id_999750650895",
"FileSourceInfos": [
{
"FileSource": "sd://opendes/autotest/test_vds_id_999750650895"
}
]
```M20 - Release 0.23Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/59Cherry-pick 'Enable user context ingestion' into release/0.212023-05-29T09:04:30ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Enable user context ingestion' into release/0.21**Original MR**: !53
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporatin...**Original MR**: !53
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/pipelines/new?ref=cherry-pick-for-53)M18 - Release 0.21David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/53Enable user context ingestion2023-05-26T16:37:55ZDeepa KumariEnable user context ingestion* ADR : https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/52
In accordance with the above ADR change involves consuming the userId passed from Airflow Conf and setting the on-behalf-of header
Related MR - ...* ADR : https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/52
In accordance with the above ADR change involves consuming the userId passed from Airflow Conf and setting the on-behalf-of header
Related MR - https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/165
User Id is a part of execution context, but is not a mandatory requirementM18 - Release 0.21Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/11feat(azure): add file handing support for azure provider2023-08-18T11:25:25ZVadzim Kulybafeat(azure): add file handing support for azure provider## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Added specific header for azure provider
Details:
https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/issues/3
Depend on:
https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/merge_requests/57M10 - Release 0.13Vadzim KulybaVadzim Kulyba