WITSML Parser merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests2023-10-11T16:15:31Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/118Draft: Update .gitlab-ci.yml2023-10-11T16:15:31Zshivani karipeDraft: Update .gitlab-ci.ymlshivani karipeshivani karipehttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/138Merge branch 'gc_fix_baremetal' into 'master'2023-10-03T14:46:05ZYan Sushchynski (EPAM)Merge branch 'gc_fix_baremetal' into 'master'This changes are for the baremetal implementation only
Update file render_dag_file.py
See merge request osdu/platform/data-flow/ingestion/energistics/witsml-parser!137
(cherry picked from commit 89fcf3a0b8b7e76fd0f99c37aaee833b0c92dec...This changes are for the baremetal implementation only
Update file render_dag_file.py
See merge request osdu/platform/data-flow/ingestion/energistics/witsml-parser!137
(cherry picked from commit 89fcf3a0b8b7e76fd0f99c37aaee833b0c92dec3)
fc40a671 Update file render_dag_file.pyM20 - Release 0.23https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/131fix azure tag pipeline2023-08-04T11:59:14Zsaketh somarajufix azure tag pipelineM19 - Release 0.22saketh somarajusaketh somarajuhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/127Draft: Az/fix az cpy dag2023-08-02T05:21:13Zsaketh somarajuDraft: Az/fix az cpy daghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/129Merge branch 'gc_fix_render_dag' into 'master'2023-08-01T19:34:15ZYan Sushchynski (EPAM)Merge branch 'gc_fix_render_dag' into 'master'gc: fix render_dag_file.py
See merge request osdu/platform/data-flow/ingestion/energistics/witsml-parser!128
(cherry picked from commit beca2db93a21b978cc17212a7b019bfccb26c136)
ca02adac gc: fix render_dag_file.pygc: fix render_dag_file.py
See merge request osdu/platform/data-flow/ingestion/energistics/witsml-parser!128
(cherry picked from commit beca2db93a21b978cc17212a7b019bfccb26c136)
ca02adac gc: fix render_dag_file.pyhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/112GONRG-6481: Refactor DAG code2023-07-11T15:38:49ZYan Sushchynski (EPAM)GONRG-6481: Refactor DAG code# Description:
Refactoring that allow run WITSML parser with different converter versions depending on provided execution context:
```
{
"executionContext": {
"Payload": {
----
},
"Context": {
...# Description:
Refactoring that allow run WITSML parser with different converter versions depending on provided execution context:
```
{
"executionContext": {
"Payload": {
----
},
"Context": {
----
"converter_version": "v2" // optional parameter that can accept "v1" or "v2" value
----
}
}
}
```
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [ ] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [ ] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [ ] Existing Tests pass
- [ ] Verified functionality locally
- [ ] Self Reviewed my code for formatting and complex business logic.Bohdan Solianyk [EPAMĀ / GCP]bohdan_solianyk@epam.comBohdan Solianyk [EPAMĀ / GCP]bohdan_solianyk@epam.comhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/107Draft: Update requirements.txt2023-02-14T11:23:34ZYan Sushchynski (EPAM)Draft: Update requirements.txthttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/87GONRG-5382: Refactor WITSML parser DAG2023-02-04T05:21:19ZYan Sushchynski (EPAM)GONRG-5382: Refactor WITSML parser DAG## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x] IBM
## Updates description?
Changes:
- Delete legacy code of uploading xml files via GCP Storage
- Make using osdu_api.ini optional in the WITSML parser image. Now we can use for instantiating Python SDK Clients the following environmental variables:
`STORAGE_SERVICE_URL`, `DATASET_SERVICE_URL`, and `STORAGE_SERVICE_URL`.M16 - Release 0.19https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/92Gonrg 5529 ivalid yaml2022-08-26T14:27:16ZMaksimelyan Tamashevich (EPAM)Gonrg 5529 ivalid yamlOleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/93cherry-pick-d3a3f618 to release/0.162022-08-24T11:54:20ZVadzim Kulybacherry-pick-d3a3f618 to release/0.16feat: upgrade base azure pipeline and added m13 support
See merge request osdu/platform/data-flow/ingestion/energistics/witsml-parser!90feat: upgrade base azure pipeline and added m13 support
See merge request osdu/platform/data-flow/ingestion/energistics/witsml-parser!90Vadzim KulybaVadzim Kulybahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/44Resolve "IBM R3M8 - Failure to ingest Wellbore data from WITSML source"2022-08-23T13:29:52ZSiarhei Khaletski (EPAM)Resolve "IBM R3M8 - Failure to ingest Wellbore data from WITSML source"NB!: Don't merge. It is just for testing.NB!: Don't merge. It is just for testing.M9 - Release 0.12etienne peyssonetienne peyssonhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/85feat: restructure azure e2e tests2022-08-19T17:12:45ZVadzim Kulybafeat: restructure azure e2e testsVadzim KulybaVadzim Kulybahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/59ibm copyright comment corrected2022-08-05T06:21:44ZGokul Nagareibm copyright comment correctedShrikant GargShrikant Garghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/78Cherry-pick 'Resolve "Code clean up"' into release/0.152022-06-17T15:19:22ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Resolve "Code clean up"' into release/0.15Original MR: !69Original MR: !69M12 - Release 0.15David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/63Upgrade First Party Library Dependencies for Release 0.142022-04-05T10:40:18ZDavid Diederichd.diederich@opengroup.orgUpgrade First Party Library Dependencies for Release 0.14This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any...This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any library that is older than the previous release will be left as-is, since the upgrade is likely to be more complicated.
Furthermore, the upgrade should only be merged in the CI pipeline reports success.
If this MR has failed, we can spend a little time investigating to see if a trivial upgrade could achieve compatiblity to the new library.
But significant upgrade efforts should not occur on this MR, as part of the release tagging process.
Instead, significant work should be scheduled for a subsequent milestone.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: 5614b7d1e887827b74bb456fa533b23911034e17
Python: 0.0.1
```
| Python Dependencies | build/providers/aws/requirements.txt | energistics/src/dags/requirements.txt | build/providers/ibm/requirements.txt | build/requirements.txt |
| ------------------- | ------------------------------------ | ------------------------------------- | ------------------------------------ | ---------------------- |
| osdu-airflow | 0.12.0 | 0.12.0 | | |
| osdu-api | 0.12.1 | 0.12.1 | 0.12.1 | 0.12.1 |
| osdu-ingestion | 0.12.0.dev133+61858640 | 0.12.0.dev133+61858640 | | 0.12.0.dev133+61858640 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade
SHA: e7a00366a706e5b575870a67a38c9f7f74248a53
Python: 0.0.1
```
| Python Dependencies | build/providers/aws/requirements.txt | energistics/src/dags/requirements.txt | build/providers/ibm/requirements.txt | build/requirements.txt |
| ------------------- | ------------------------------------ | ------------------------------------- | ------------------------------------ | ---------------------- |
| osdu-airflow | 0.12.0 | 0.12.0 | | |
| osdu-api | 0.12.1 | 0.12.1 | 0.12.1 | 0.12.1 |
| osdu-ingestion | 0.12.0.dev133+61858640 | 0.12.0.dev133+61858640 | | 0.12.0.dev133+61858640 |M11 - Release 0.14https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/43WIP: GONRG-3779: Common pipeline for dag2021-12-20T14:42:32ZYan Sushchynski (EPAM)WIP: GONRG-3779: Common pipeline for dagAdd common ppipelines for GCPAdd common ppipelines for GCPM10 - Release 0.13Siarhei Khaletski (EPAM)Vladislav Shishko (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/20ibm witsml changes2021-07-27T12:28:10ZGokul Nagareibm witsml changeshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics-osdu-integration/-/issues/38https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics-osdu-integration/-/issues/38Anuj GuptaAnuj Guptahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/10GONRG-2106: Docker image for different providers2021-04-07T13:36:11ZYan Sushchynski (EPAM)GONRG-2106: Docker image for different providersThe main goal of the MR is to create an approach for building WITSML parser image for every coud-provider.The main goal of the MR is to create an approach for building WITSML parser image for every coud-provider.Laurent DenyLaurent Denyhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/5Deploy on GCP Composer Airflow (GONRG-888)2020-10-27T21:56:58ZYan Sushchynski (EPAM)Deploy on GCP Composer Airflow (GONRG-888)Main changes:
1. Cloud Composer can't properly process file with code in the `plugins` folder if these files are big, as it is in our case. So the libs folder was moved into the `dags` folder. If we don't want Airflow to search DAGs in ...Main changes:
1. Cloud Composer can't properly process file with code in the `plugins` folder if these files are big, as it is in our case. So the libs folder was moved into the `dags` folder. If we don't want Airflow to search DAGs in the libs folder, we must add the `.airflowignore` into energistics/dags with specifying files we want to ignore.
2. Add an extra steps to make File part of Manifests follow R2 schema in process_energistics_op.py.
3. Some minor bug fixes in process_energistics_op.py.Mohammed Alimohammed.ali@energistics.orgMohammed Alimohammed.ali@energistics.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/4Deploy on GCP Composer Airflow (GONRG-888)2020-10-07T07:19:36ZYan Sushchynski (EPAM)Deploy on GCP Composer Airflow (GONRG-888)Main changes:
1. Cloud Composer can't properly process file with code in the plugins folder if these files are big, as it is in our case. So the libs folder was moved into the dags folder.
2. Add an extra steps to make File part of Man...Main changes:
1. Cloud Composer can't properly process file with code in the plugins folder if these files are big, as it is in our case. So the libs folder was moved into the dags folder.
2. Add an extra steps to make File part of Manifests follow R2 schema in process_energistics_op.py.
3. Fix `convert_to_r2_schemas` in process_energistics_op.py.
4. Some minor bug fixes in process_energistics_op.py.