Ingestion Workflow merge requests
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests
2024-03-20T09:04:02Z
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/531
java17migration
2024-03-20T09:04:02Z
Vikas Hoode [BP]
vikas.hoode@bp.com
java17migration
Jdk 17 migration
Jdk 17 migration
M23 - Release 0.26
Vikas Hoode [BP]
vikas.hoode@bp.com
Vikas Hoode [BP]
vikas.hoode@bp.com
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/513
Merge branch 'api-extend-run-details' into 'master'
2024-01-05T09:32:59Z
Riabokon Stanislav(EPAM)[GCP]
Merge branch 'api-extend-run-details' into 'master'
Implement Airflow facade endpoint
See merge request osdu/platform/data-flow/ingestion/ingestion-workflow!505
(cherry picked from commit 2464a39b22f5dc9c984db45fe09b361409cefd93)
a26233b7 added api endpoint to fetch task details with x...
Implement Airflow facade endpoint
See merge request osdu/platform/data-flow/ingestion/ingestion-workflow!505
(cherry picked from commit 2464a39b22f5dc9c984db45fe09b361409cefd93)
a26233b7 added api endpoint to fetch task details with xcom values
69dc8125 undo Stas test changes
ba6417ef move tests
61cb1746 Merge remote-tracking branch 'origin/master' into api-extend-run-details
fa5847d1 Merge remote-tracking branch 'origin/master' into api-extend-run-details
502f760e Merge remote-tracking branch 'origin/master' into api-extend-run-details
2f44c1c8 Added condition to /{workflow_name}/workflowRun/{runId}/latestInfo
M22 - Release 0.25
David Diederich
d.diederich@opengroup.org
David Diederich
d.diederich@opengroup.org
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/481
Pass workflow user ID to the Airflow as part of payload (GONRG-7795)
2023-11-08T10:37:23Z
Riabokon Stanislav(EPAM)[GCP]
Pass workflow user ID to the Airflow as part of payload (GONRG-7795)
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/157
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/158
Removed a custom header 'x-user-id' fro...
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/157
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/158
Removed a custom header 'x-user-id' from a core part of service.
M22 - Release 0.25
Riabokon Stanislav(EPAM)[GCP]
Riabokon Stanislav(EPAM)[GCP]
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/401
Full Upgrade of First Party Library Dependencies for Release 0.20
2023-05-22T15:17:48Z
David Diederich
d.diederich@opengroup.org
Full Upgrade of First Party Library Dependencies for Release 0.20
This generated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to try to fully upgrade all dependent libraries to see if the latest code will work.
It is expected that these will ...
This generated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to try to fully upgrade all dependent libraries to see if the latest code will work.
It is expected that these will often fail, since the upgrades were previously rejected for failing pipelines and have not been directly addressed yet.
This upgrade should only be merged in the CI pipeline reports success.
If this MR has failed, we can spend a little time investigating to see if a trivial upgrade could achieve compatiblity to the new library.
But significant upgrade efforts should not occur on this MR, as part of the release tagging process.
Instead, significant work should be scheduled for a subsequent milestone.
This MR may co-exist with a separate, smaller upgrade MR.
If both pass, this one should be used instead.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: ad4e2951c9d4004fb28604322f95c742906dbbb8
Maven: 0.21.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | ---------------------------------------------- | --------------------- |
| core-lib-azure | 0.14.0-rc2 | 0.0.28 |
| core-lib-gcp | 0.19.0-rc3 | |
| os-core-lib-aws | 0.19.0-rc3 | 0.14.0-rc2 |
| obm | 0.18.0 | |
| oqm | 0.18.0 | |
| os-core-common | 0.19.0-rc8 | 0.13.0 |
| os-core-lib-ibm | 0.16.0-rc1 | 0.8.0 |
| osm | 0.18.0 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.10.5, 2.13.2.2, 2.13.4.2 | 2.13.2.2, 2.10.1 |
| (3rd Party) org.apache.logging.log4j.log4j-core | 2.17.1 | 2.12.1 |
| (3rd Party) org.apache.logging.log4j.log4j-jul | 2.17.1 | 2.12.1 |
| (3rd Party) org.apache.logging.log4j.log4j-slf4j-impl | 2.17.1 | 2.12.1 |
| (3rd Party) org.springframework.spring-webmvc | 5.2.10.RELEASE, 5.2.22.RELEASE, 5.3.24, 5.3.22 | 5.2.2.RELEASE, 5.3.12 |
| (3rd Party) org.yaml.snakeyaml | 1.25, 2.0 | 1.25, 1.27 |
```
Critical: Found Vulnerable Spring MVC dependency (<5.2.20 || >=5.3.0 <5.3.18)
├─ _Root_
│ ├─ io.springfox.springfox-boot-starter == 3.0.0
│ │ └─ org.opengroup.osdu.os-core-common == 0.19.0-rc8
│ │ └─ org.springframework.spring-webmvc == 5.2.10.RELEASE
│ └─ org.opengroup.osdu.workflow-ibm == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.workflow-core == 0.21.0-SNAPSHOT
│ └─ org.springframework.spring-webmvc == 5.2.10.RELEASE
└─ testing/
├─ org.opengroup.osdu.workflow-test-core == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.13.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.2.2.RELEASE
│ └─ org.springframework.spring-webmvc == 5.2.2.RELEASE
├─ org.opengroup.osdu.workflow.workflow-test-aws == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.core.aws.os-core-lib-aws == 0.14.0-rc2
│ └─ org.opengroup.osdu.os-core-common == 0.13.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.2.2.RELEASE
│ └─ org.springframework.spring-webmvc == 5.2.2.RELEASE
├─ org.opengroup.osdu.workflow-test-gc == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.workflow-test-core == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.13.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.2.2.RELEASE
│ └─ org.springframework.spring-webmvc == 5.2.2.RELEASE
├─ org.opengroup.osdu.workflow.workflow-test-azure == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.13.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.4.12
│ └─ org.springframework.spring-webmvc == 5.3.12
├─ org.opengroup.osdu.workflow-test-ibm == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.workflow-test-core == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.13.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.2.2.RELEASE
│ └─ org.springframework.spring-webmvc == 5.2.2.RELEASE
└─ org.opengroup.osdu.workflow-test-anthos == 0.21.0-SNAPSHOT
└─ org.opengroup.osdu.workflow-test-core == 0.21.0-SNAPSHOT
└─ org.opengroup.osdu.os-core-common == 0.13.0
└─ org.springframework.boot.spring-boot-starter-web == 2.2.2.RELEASE
└─ org.springframework.spring-webmvc == 5.2.2.RELEASE
```
```
Critical: Found Vulnerable Snake YAML dependency (<2.0)
├─ _Root_
│ ├─ io.springfox.springfox-boot-starter == 3.0.0
│ │ └─ org.springframework.cloud.spring-cloud-starter == 2.2.2.RELEASE
│ │ └─ org.springframework.boot.spring-boot-starter == 2.2.11.RELEASE
│ │ └─ org.yaml.snakeyaml == 1.25
│ ├─ org.opengroup.osdu.workflow-aws == 0.21.0-SNAPSHOT
│ │ └─ org.opengroup.osdu.workflow-core == 0.21.0-SNAPSHOT
│ │ └─ org.yaml.snakeyaml == 1.25
│ └─ org.opengroup.osdu.workflow-gc == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.workflow-core == 0.21.0-SNAPSHOT
│ └─ org.yaml.snakeyaml == 1.25
└─ testing/
├─ org.opengroup.osdu.workflow-test-core == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.13.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.2.2.RELEASE
│ └─ org.springframework.boot.spring-boot-starter == 2.2.2.RELEASE
│ └─ org.yaml.snakeyaml == 1.25
├─ org.opengroup.osdu.workflow.workflow-test-aws == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.core.aws.os-core-lib-aws == 0.14.0-rc2
│ └─ org.opengroup.osdu.os-core-common == 0.13.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.2.2.RELEASE
│ └─ org.springframework.boot.spring-boot-starter == 2.2.2.RELEASE
│ └─ org.yaml.snakeyaml == 1.25
├─ org.opengroup.osdu.workflow-test-gc == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.workflow-test-core == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.13.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.2.2.RELEASE
│ └─ org.springframework.boot.spring-boot-starter == 2.2.2.RELEASE
│ └─ org.yaml.snakeyaml == 1.25
├─ org.opengroup.osdu.workflow.workflow-test-azure == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.13.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.4.12
│ └─ org.springframework.boot.spring-boot-starter == 2.4.12
│ └─ org.yaml.snakeyaml == 1.27
├─ org.opengroup.osdu.workflow-test-ibm == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-lib-ibm == 0.8.0
│ └─ org.springframework.boot.spring-boot-starter-security == 2.2.2.RELEASE
│ └─ org.springframework.boot.spring-boot-starter == 2.2.2.RELEASE
│ └─ org.yaml.snakeyaml == 1.25
└─ org.opengroup.osdu.workflow-test-anthos == 0.21.0-SNAPSHOT
└─ org.opengroup.osdu.workflow-test-core == 0.21.0-SNAPSHOT
└─ org.opengroup.osdu.os-core-common == 0.13.0
└─ org.springframework.boot.spring-boot-starter-web == 2.2.2.RELEASE
└─ org.springframework.boot.spring-boot-starter == 2.2.2.RELEASE
└─ org.yaml.snakeyaml == 1.25
```
```
Critical: Found Vulnerable Jackson Databind dependency (<2.12.6.1 || >=2.13.0 <2.13.2.1)
├─ _Root_
│ └─ io.springfox.springfox-boot-starter == 3.0.0
│ └─ com.fasterxml.jackson.core.jackson-databind == 2.10.5
└─ testing/
├─ org.opengroup.osdu.workflow-test-gc == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.workflow-test-core == 0.21.0-SNAPSHOT
│ └─ com.fasterxml.jackson.core.jackson-databind == 2.10.1
├─ org.opengroup.osdu.workflow.workflow-test-azure == 0.21.0-SNAPSHOT
│ └─ com.fasterxml.jackson.core.jackson-databind == 2.10.1
├─ org.opengroup.osdu.workflow-test-ibm == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.workflow-test-core == 0.21.0-SNAPSHOT
│ └─ com.fasterxml.jackson.core.jackson-databind == 2.10.1
└─ org.opengroup.osdu.workflow-test-anthos == 0.21.0-SNAPSHOT
└─ org.opengroup.osdu.workflow-test-core == 0.21.0-SNAPSHOT
└─ com.fasterxml.jackson.core.jackson-databind == 2.10.1
```
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade
SHA: da526b77c99d93cb37e06a34420722ff6652514a
Maven: 0.21.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | ---------------------------------------------- | --------------------- |
| core-lib-azure | 0.20.0 | 0.20.0 |
| core-lib-gc | 0.20.0 | |
| os-core-lib-aws | 0.20.0 | 0.20.0 |
| obm | 0.20.0 | |
| oqm | 0.20.0 | |
| os-core-common | 0.20.1 | 0.20.1 |
| os-core-lib-ibm | 0.20.0 | 0.20.0 |
| osm | 0.20.0 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.10.5, 2.13.2.2, 2.13.4.2 | 2.13.2.2, 2.10.1 |
| (3rd Party) org.springframework.spring-webmvc | 5.2.10.RELEASE, 5.2.22.RELEASE, 5.3.24, 5.3.22 | 5.2.2.RELEASE, 5.3.22 |
| (3rd Party) org.yaml.snakeyaml | 1.25, 2.0 | 1.25, 1.27 |
```
Critical: Found Vulnerable Spring MVC dependency (<5.2.20 || >=5.3.0 <5.3.18)
├─ _Root_
│ ├─ io.springfox.springfox-boot-starter == 3.0.0
│ │ └─ org.opengroup.osdu.os-core-common == 0.20.1
│ │ └─ org.springframework.spring-webmvc == 5.2.10.RELEASE
│ └─ org.opengroup.osdu.workflow-ibm == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.workflow-core == 0.21.0-SNAPSHOT
│ └─ org.springframework.spring-webmvc == 5.2.10.RELEASE
└─ testing/
├─ org.opengroup.osdu.workflow-test-core == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.20.1
│ └─ org.springframework.spring-webmvc == 5.2.2.RELEASE
├─ org.opengroup.osdu.workflow.workflow-test-aws == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.core.aws.os-core-lib-aws == 0.20.0
│ └─ org.springframework.spring-webmvc == 5.2.2.RELEASE
├─ org.opengroup.osdu.workflow-test-gc == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.workflow-test-core == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.20.1
│ └─ org.springframework.spring-webmvc == 5.2.2.RELEASE
├─ org.opengroup.osdu.workflow-test-ibm == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.workflow-test-core == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.20.1
│ └─ org.springframework.spring-webmvc == 5.2.2.RELEASE
└─ org.opengroup.osdu.workflow-test-anthos == 0.21.0-SNAPSHOT
└─ org.opengroup.osdu.workflow-test-core == 0.21.0-SNAPSHOT
└─ org.opengroup.osdu.os-core-common == 0.20.1
└─ org.springframework.spring-webmvc == 5.2.2.RELEASE
```
```
Critical: Found Vulnerable Snake YAML dependency (<2.0)
├─ _Root_
│ ├─ io.springfox.springfox-boot-starter == 3.0.0
│ │ └─ org.springframework.cloud.spring-cloud-starter == 2.2.2.RELEASE
│ │ └─ org.springframework.boot.spring-boot-starter == 2.2.11.RELEASE
│ │ └─ org.yaml.snakeyaml == 1.25
│ ├─ org.opengroup.osdu.workflow-aws == 0.21.0-SNAPSHOT
│ │ └─ org.opengroup.osdu.workflow-core == 0.21.0-SNAPSHOT
│ │ └─ org.yaml.snakeyaml == 1.25
│ └─ org.opengroup.osdu.workflow-gc == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.workflow-core == 0.21.0-SNAPSHOT
│ └─ org.yaml.snakeyaml == 1.25
└─ testing/
├─ org.opengroup.osdu.workflow-test-core == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.20.1
│ └─ org.springframework.boot.spring-boot-starter-web == 2.2.2.RELEASE
│ └─ org.springframework.boot.spring-boot-starter == 2.2.2.RELEASE
│ └─ org.yaml.snakeyaml == 1.25
├─ org.opengroup.osdu.workflow.workflow-test-aws == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.core.aws.os-core-lib-aws == 0.20.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.2.2.RELEASE
│ └─ org.springframework.boot.spring-boot-starter == 2.2.2.RELEASE
│ └─ org.yaml.snakeyaml == 1.25
├─ org.opengroup.osdu.workflow-test-gc == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.workflow-test-core == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.20.1
│ └─ org.springframework.boot.spring-boot-starter-web == 2.2.2.RELEASE
│ └─ org.springframework.boot.spring-boot-starter == 2.2.2.RELEASE
│ └─ org.yaml.snakeyaml == 1.25
├─ org.opengroup.osdu.workflow.workflow-test-azure == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.core-lib-azure == 0.20.0
│ └─ org.redisson.redisson == 3.15.3
│ └─ org.yaml.snakeyaml == 1.27
├─ org.opengroup.osdu.workflow-test-ibm == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-lib-ibm == 0.20.0
│ └─ org.yaml.snakeyaml == 1.25
└─ org.opengroup.osdu.workflow-test-anthos == 0.21.0-SNAPSHOT
└─ org.opengroup.osdu.workflow-test-core == 0.21.0-SNAPSHOT
└─ org.opengroup.osdu.os-core-common == 0.20.1
└─ org.springframework.boot.spring-boot-starter-web == 2.2.2.RELEASE
└─ org.springframework.boot.spring-boot-starter == 2.2.2.RELEASE
└─ org.yaml.snakeyaml == 1.25
```
```
Critical: Found Vulnerable Jackson Databind dependency (<2.12.6.1 || >=2.13.0 <2.13.2.1)
├─ _Root_
│ └─ io.springfox.springfox-boot-starter == 3.0.0
│ └─ com.fasterxml.jackson.core.jackson-databind == 2.10.5
└─ testing/
├─ org.opengroup.osdu.workflow-test-gc == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.workflow-test-core == 0.21.0-SNAPSHOT
│ └─ com.fasterxml.jackson.core.jackson-databind == 2.10.1
├─ org.opengroup.osdu.workflow.workflow-test-azure == 0.21.0-SNAPSHOT
│ └─ com.fasterxml.jackson.core.jackson-databind == 2.10.1
├─ org.opengroup.osdu.workflow-test-ibm == 0.21.0-SNAPSHOT
│ └─ org.opengroup.osdu.workflow-test-core == 0.21.0-SNAPSHOT
│ └─ com.fasterxml.jackson.core.jackson-databind == 2.10.1
└─ org.opengroup.osdu.workflow-test-anthos == 0.21.0-SNAPSHOT
└─ org.opengroup.osdu.workflow-test-core == 0.21.0-SNAPSHOT
└─ com.fasterxml.jackson.core.jackson-databind == 2.10.1
```
M18 - Release 0.21
David Diederich
d.diederich@opengroup.org
Srinivasan Narayanan
David Diederich
d.diederich@opengroup.org
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/368
jar type vulnerability fix for ingestion-workflow-service
2023-03-30T11:26:01Z
Pintu Gupta
jar type vulnerability fix for ingestion-workflow-service
Following CVE has been fix in this MR :
| cve | link |
|------------------|-------------------------------------------------|
| CVE-2022-25857 | https://nvd.nist.gov/vuln/detail/...
Following CVE has been fix in this MR :
| cve | link |
|------------------|-------------------------------------------------|
| CVE-2022-25857 | https://nvd.nist.gov/vuln/detail/CVE-2022-25857 |
| CVE-2021-22118 | https://nvd.nist.gov/vuln/detail/CVE-2021-22118 |
| PRISMA-2022-0239 | https://github.com/square/okhttp/issues/6738 |
| CVE-2022-42252 | https://nvd.nist.gov/vuln/detail/CVE-2022-42252 |
| CVE-2022-22965 | https://nvd.nist.gov/vuln/detail/CVE-2022-22965 |
| CVE-2022-25647 | https://nvd.nist.gov/vuln/detail/CVE-2022-25647 |
| CVE-2022-22965 | https://nvd.nist.gov/vuln/detail/CVE-2022-22965 |
| CVE-2022-31692 | https://nvd.nist.gov/vuln/detail/CVE-2022-31692 |
| CVE-2022-22965 | https://nvd.nist.gov/vuln/detail/CVE-2022-22965 |
| CVE-2021-22118 | https://nvd.nist.gov/vuln/detail/CVE-2021-22118 |
| CVE-2022-22968 | https://nvd.nist.gov/vuln/detail/CVE-2022-22968 |
| CVE-2022-22970 | https://nvd.nist.gov/vuln/detail/CVE-2022-22970 |
| CVE-2022-42003 | https://nvd.nist.gov/vuln/detail/CVE-2022-42003 |
| CVE-2022-42004 | https://nvd.nist.gov/vuln/detail/CVE-2022-42004 |
M17 - Release 0.20
Pintu Gupta
Pintu Gupta
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/363
Bugfix: fix misleading logs
2023-05-22T07:22:38Z
Maksim Malkov
Bugfix: fix misleading logs
## Changes
- change log level for messages which printed from repository level during metadata search in both system and partition DBs
- update searching metadata logic accordingly (we do not need exception based branching here)
- update...
## Changes
- change log level for messages which printed from repository level during metadata search in both system and partition DBs
- update searching metadata logic accordingly (we do not need exception based branching here)
- update unit and component tests
[issue link](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/150)
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/313
Pass user id as part of airflow conf
2023-02-04T05:19:54Z
harshit aggarwal
Pass user id as part of airflow conf
Related ADR - https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/52
In accordance with the above ADR change involves adding the userId field into execution context in Airflow Conf Object
Related ADR - https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/52
In accordance with the above ADR change involves adding the userId field into execution context in Airflow Conf Object
M16 - Release 0.19
harshit aggarwal
harshit aggarwal
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/285
Make data-partition-id parameter optional for system endpoints
2023-02-04T05:19:58Z
Aliaksei Kalachou
Make data-partition-id parameter optional for system endpoints
System workflow endpoints in workflow service are to manage(create/delete) system workflows which are available to all data partitions by default. Hence system endpoints doesn't expect and data partition id but still in swagger it wrongl...
System workflow endpoints in workflow service are to manage(create/delete) system workflows which are available to all data partitions by default. Hence system endpoints doesn't expect and data partition id but still in swagger it wrongly mentioned as mandatory. This MR intend to make data partition is as non mandatory for system endpoints in swagger. No functional enhancement or modification in behavior of workflow service.
M16 - Release 0.19
Aliaksei Kalachou
Aliaksei Kalachou
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/265
Do not mark a workflow as 'failed' if the status is set to 'finished'
2022-04-18T18:24:38Z
Morris Estepa
Do not mark a workflow as 'failed' if the status is set to 'finished'
commit 4bf7df95
Author: Morris Estepa <estepamo@amazon.com>
Date: Wed Apr 13 2022 16:49:37 GMT-0500 (Central Daylight Time)
Revert "Change finished to success when the status passed to the service is finished."
This reverts comm...
commit 4bf7df95
Author: Morris Estepa <estepamo@amazon.com>
Date: Wed Apr 13 2022 16:49:37 GMT-0500 (Central Daylight Time)
Revert "Change finished to success when the status passed to the service is finished."
This reverts commit 6bc84f170260675f76428f3424ecb6b544fa4f01.
commit 6bc84f17
Author: Morris Estepa <estepamo@amazon.com>
Date: Wed Apr 13 2022 16:14:35 GMT-0500 (Central Daylight Time)
Change finished to success when the status passed to the service is finished.
commit 90bd0994
Author: Morris Estepa <estepamo@amazon.com>
Date: Wed Apr 13 2022 14:15:10 GMT-0500 (Central Daylight Time)
Remove unused vars; add copyright.
commit b6db3862
Author: Morris Estepa <estepamo@amazon.com>
Date: Tue Apr 12 2022 16:59:38 GMT-0500 (Central Daylight Time)
Implement GSM
Morris Estepa
Morris Estepa
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/202
Add airflow version for info endpoint, enable stable API for gcp (GONRG-3777)
2021-11-29T13:34:53Z
Rustam Lotsmanenko (EPAM)
rustam_lotsmanenko@epam.com
Add airflow version for info endpoint, enable stable API for gcp (GONRG-3777)
# Description:
Issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/132 <br/>
Add a version of Airflow into an endpoint 'info' for Workflow Service.<br/>
A version of Airflow will be reques...
# Description:
Issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/132 <br/>
Add a version of Airflow into an endpoint 'info' for Workflow Service.<br/>
A version of Airflow will be requested only when enabled stable API `osdu.airflow.version2=true`.<br/>
Enable stable API for gcp.
# How to test:
Functionality can be testes with a request:
~~~
curl --location --request GET 'https://com.osdu.club/api/workflow/info' \
~~~
Response example:
~~~
{
"groupId": "org.opengroup.osdu",
"artifactId": "workflow-gcp",
"version": "0.13.0-SNAPSHOT",
"buildTime": "2021-11-25T12:29:41.581Z",
"branch": "add-airflow-version-for-info-endpoint",
"commitId": "5e0816ea1088e6205fae839db389ebf403dc7885",
"commitMessage": "Add airflow version for info endpoint, enable stable API for gcp",
"connectedOuterServices": [
{
"name": "Airflow",
"version": "2.1.2+composer"
}
]
}
~~~
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [x] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
- [x] Common code
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.
M10 - Release 0.13
Riabokon Stanislav(EPAM)[GCP]
Riabokon Stanislav(EPAM)[GCP]
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/198
Stabilize Workflow with postman collection (new Approach) (GONRG-3729)
2021-11-19T12:27:13Z
Artem Dobrynin (EPAM)
Stabilize Workflow with postman collection (new Approach) (GONRG-3729)
# Description:
There was an issue while creating system workflow entities. The flag has been ignored, therefore, some logic for system workflow has been skipping.
Also, there was some miswritten exception for workflow not found
# How t...
# Description:
There was an issue while creating system workflow entities. The flag has been ignored, therefore, some logic for system workflow has been skipping.
Also, there was some miswritten exception for workflow not found
# How to test:
Functionality was tested on GCP with datastore and postgres
# Changes include:
* [ ] Refactor (a non-breaking change that improves code maintainability).
* [X] Bugfix (a non-breaking change that solves an issue).
* [ ] New feature (a non-breaking change that adds functionality).
* [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
* [x] GCP
* [ ] Azure
* [ ] AWS
* [ ] IBM
* [x] Common code
# Dev Checklist:
* [x] Added Unit Tests, wherever applicable.
* [x] Updated the Readme, if applicable.
* [x] Existing Tests pass
* [x] Verified functionality locally
* [x] Self Reviewed my code for formatting and complex business logic.
# Other comments:
Should be merged after https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/191
M10 - Release 0.13
Riabokon Stanislav(EPAM)[GCP]
Rostislav Dublin (EPAM)
Riabokon Stanislav(EPAM)[GCP]
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/127
500 error in Workflow with incorrect request body on Preshipping (GONRG-2499)
2021-06-22T20:28:10Z
Artem Dobrynin (EPAM)
500 error in Workflow with incorrect request body on Preshipping (GONRG-2499)
# Description:
Added exception handling for JSON mapping
# How to test:
Ingestion Workflow service can be tested with the help of business cases.
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainabili...
# Description:
Added exception handling for JSON mapping
# How to test:
Ingestion Workflow service can be tested with the help of business cases.
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [x] Bugfix (a non-breaking change that solves an issue).
- [ ] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [ ] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
- [x] Core
# Dev Checklist:
* [x] Added Unit Tests, wherever applicable.
* [ ] Updated the Readme, if applicable.
* [x] Existing Tests pass
* [x] Verified functionality locally
* [x] Self Reviewed my code for formatting and complex business logic.
Rostislav Dublin (EPAM)
Rostislav Dublin (EPAM)
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/109
[Core] Added Integration Tests
2021-08-03T07:14:49Z
Aalekh Jain
[Core] Added Integration Tests