Ingestion Workflow merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests2023-08-18T11:18:27Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/165Disable accepting dag content based on feature flag [azure]2023-08-18T11:18:27ZAalekh JainDisable accepting dag content based on feature flag [azure]M10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/182Gcp workflow osm(GONRG-3404)2021-11-03T09:23:18ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comGcp workflow osm(GONRG-3404)# Description:
Implementation of new common code design based on OSM framework.
# How to test:
Functionality was tested with Datastore and Postgres integration tests passed.
# Changes include:
- [x] Refactor (a non-breaking change that...# Description:
Implementation of new common code design based on OSM framework.
# How to test:
Functionality was tested with Datastore and Postgres integration tests passed.
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [x] New feature (a non-breaking change that adds functionality).
# Changes in:
- [x] GCP
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [x] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.
# Other comments:
Migration cannot be done in one step, wanting to keep the current data structure (not model native),
this MR includes several classes that can be removed later:
~~~
`@ConditionalOnProperty(name = "datastore.legacy.data.structure", havingValue = "true")`
~~~
OSM framework is currently **not published** to the community, and the package registry does not have an artifact of it, the build **will fail** until OSM is published.M10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Rostislav Dublin (EPAM)Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/191Added iaap auth for airflow stable api (GONRG-3753)2021-11-19T11:36:12ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comAdded iaap auth for airflow stable api (GONRG-3753)# Description:
Refactor workflow to use different approaches for auth with Airflow 2.0.
This [MR](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/160) brings Airflow 2.0 stable API to...# Description:
Refactor workflow to use different approaches for auth with Airflow 2.0.
This [MR](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/160) brings Airflow 2.0 stable API to workflow service, but it has hardcoded Basic auth as the only way to authenticate requests to Airflow. We need to un-hardcode authentication and verify work with Airflow 2.0 stable API.
Airflow2 can be deployed either in Composer or K8S.
If it is deployed in Composer, then Authorization remains the same as it was before (Bearer: <IAAP_Token>), because Composer uses its own auth backend for Stable API
If Airflow is deployed in K8S, then we can use Basic Auth.
Workflow Service must have the possibility to choose one of these two options depending on where AF2 is deployed.
In order to switch between `stable` and `experimental` API env variable `OSDU_AIRFLOW_VERSION2` may be used.<br/>
In order to switch between `basic` and `IAAP` auth env variable `AIRFLOW_IAAP_MODE` may be used.
# How to test:
Functionality was tested with `experimental` and `stable` API
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [x] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
- [ ] Common code
# Dev Checklist:
- [x] Added Unit Tests, wherever applicable.
- [x] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Rostislav Dublin (EPAM)Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/193Update .gitlab-ci.yml2023-08-18T11:18:06ZAnuj GuptaUpdate .gitlab-ci.ymlM10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/194Airflow2 stable api support2023-08-18T11:18:04ZSaurabh TripathiAirflow2 stable api support### Summary of changes
* Make changes to the connect to airflow2 stable API and accommodate the response changes
* UT for the airflow2 implementations
### Related changes
* add airflow env selection variable in pipeline
* Documentation### Summary of changes
* Make changes to the connect to airflow2 stable API and accommodate the response changes
* UT for the airflow2 implementations
### Related changes
* add airflow env selection variable in pipeline
* DocumentationM10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/197[Azure] Add support for caching in workflow ingestion service2023-08-18T11:18:02ZAalekh Jain[Azure] Add support for caching in workflow ingestion serviceM10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/198Stabilize Workflow with postman collection (new Approach) (GONRG-3729)2021-11-19T12:27:13ZArtem Dobrynin (EPAM)Stabilize Workflow with postman collection (new Approach) (GONRG-3729)# Description:
There was an issue while creating system workflow entities. The flag has been ignored, therefore, some logic for system workflow has been skipping.
Also, there was some miswritten exception for workflow not found
# How t...# Description:
There was an issue while creating system workflow entities. The flag has been ignored, therefore, some logic for system workflow has been skipping.
Also, there was some miswritten exception for workflow not found
# How to test:
Functionality was tested on GCP with datastore and postgres
# Changes include:
* [ ] Refactor (a non-breaking change that improves code maintainability).
* [X] Bugfix (a non-breaking change that solves an issue).
* [ ] New feature (a non-breaking change that adds functionality).
* [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
* [x] GCP
* [ ] Azure
* [ ] AWS
* [ ] IBM
* [x] Common code
# Dev Checklist:
* [x] Added Unit Tests, wherever applicable.
* [x] Updated the Readme, if applicable.
* [x] Existing Tests pass
* [x] Verified functionality locally
* [x] Self Reviewed my code for formatting and complex business logic.
# Other comments:
Should be merged after https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/191M10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Rostislav Dublin (EPAM)Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/199[Azure] Limiting trigger workflow requests in workflow ingestion service2023-08-18T11:18:00ZAalekh Jain[Azure] Limiting trigger workflow requests in workflow ingestion serviceM10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/200Stabilize Workflow with postman collection (new Approach) (GONRG-3729)2023-08-18T11:17:59ZArtem Dobrynin (EPAM)Stabilize Workflow with postman collection (new Approach) (GONRG-3729)# Description:
There was an issue while creating system workflow entities. The flag has been ignored, therefore, some logic for system workflow has been skipping.
Also, there was some miswritten exception for workflow not found
# How t...# Description:
There was an issue while creating system workflow entities. The flag has been ignored, therefore, some logic for system workflow has been skipping.
Also, there was some miswritten exception for workflow not found
# How to test:
Functionality was tested on GCP with datastore and postgres
# Changes include:
* [ ] Refactor (a non-breaking change that improves code maintainability).
* [X] Bugfix (a non-breaking change that solves an issue).
* [ ] New feature (a non-breaking change that adds functionality).
* [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
* [x] GCP
* [ ] Azure
* [ ] AWS
* [ ] IBM
* [x] Common code
# Dev Checklist:
* [x] Added Unit Tests, wherever applicable.
* [x] Updated the Readme, if applicable.
* [x] Existing Tests pass
* [x] Verified functionality locally
* [x] Self Reviewed my code for formatting and complex business logic.
# Other comments:M10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Rostislav Dublin (EPAM)Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/201Disable spring web security 2052023-08-18T11:17:57ZBhushan RadeDisable spring web security 205M10 - Release 0.13Anuj GuptaBhushan RadeAnuj Guptahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/202Add airflow version for info endpoint, enable stable API for gcp (GONRG-3777)2021-11-29T13:34:53ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comAdd airflow version for info endpoint, enable stable API for gcp (GONRG-3777)# Description:
Issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/132 <br/>
Add a version of Airflow into an endpoint 'info' for Workflow Service.<br/>
A version of Airflow will be reques...# Description:
Issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/132 <br/>
Add a version of Airflow into an endpoint 'info' for Workflow Service.<br/>
A version of Airflow will be requested only when enabled stable API `osdu.airflow.version2=true`.<br/>
Enable stable API for gcp.
# How to test:
Functionality can be testes with a request:
~~~
curl --location --request GET 'https://com.osdu.club/api/workflow/info' \
~~~
Response example:
~~~
{
"groupId": "org.opengroup.osdu",
"artifactId": "workflow-gcp",
"version": "0.13.0-SNAPSHOT",
"buildTime": "2021-11-25T12:29:41.581Z",
"branch": "add-airflow-version-for-info-endpoint",
"commitId": "5e0816ea1088e6205fae839db389ebf403dc7885",
"commitMessage": "Add airflow version for info endpoint, enable stable API for gcp",
"connectedOuterServices": [
{
"name": "Airflow",
"version": "2.1.2+composer"
}
]
}
~~~
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [x] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
- [x] Common code
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/203Enable airflow stable api for gcp2021-11-29T15:31:55ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comEnable airflow stable api for gcp# Description:
Enable stable API for gcp. Minor refactoring & unhardcode integration tests.
~~~
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solv...# Description:
Enable stable API for gcp. Minor refactoring & unhardcode integration tests.
~~~
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [x] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/204Add a version of Airflow into an endpoint 'info' for Workflow Service(GONRG-3...2021-12-15T19:43:19ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comAdd a version of Airflow into an endpoint 'info' for Workflow Service(GONRG-3777)# Description:
Issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/132 <br/>
Add a version of Airflow into an endpoint 'info' for Workflow Service.<br/>
A version of Airflow will be reques...# Description:
Issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/132 <br/>
Add a version of Airflow into an endpoint 'info' for Workflow Service.<br/>
A version of Airflow will be requested only when enabled stable API `osdu.airflow.version2=true`.<br/>
# How to test:
Functionality can be testes with a request:
~~~
curl --location --request GET 'https://com.osdu.club/api/workflow/info' \
~~~
Response example:
~~~
{
"groupId": "org.opengroup.osdu",
"artifactId": "workflow-gcp",
"version": "0.13.0-SNAPSHOT",
"buildTime": "2021-11-25T12:29:41.581Z",
"branch": "add-airflow-version-for-info-endpoint",
"commitId": "5e0816ea1088e6205fae839db389ebf403dc7885",
"commitMessage": "Add airflow version for info endpoint, enable stable API for gcp",
"connectedOuterServices": [
{
"name": "Airflow",
"version": "2.1.2+composer"
}
]
}
~~~
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [x] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [ ] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
- [x] Common code
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/207Vuln fix lucene2023-08-18T11:17:55ZGokul NagareVuln fix luceneM10 - Release 0.13Anuj GuptaShrikant GargAnuj Guptahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/208log4j vuln fix2023-08-18T11:17:54ZBhushan Radelog4j vuln fix|module pom changes| Ref Issue|
|------|-----------------------------------------------------------------------------------------------|
| IBM | https://community.opengroup.org/osdu/platform/system/lib/cloud/ibm/os-core-lib-ibm/-/issues...|module pom changes| Ref Issue|
|------|-----------------------------------------------------------------------------------------------|
| IBM | https://community.opengroup.org/osdu/platform/system/lib/cloud/ibm/os-core-lib-ibm/-/issues/2 |
| Core | https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/54 |
Part of https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/134 seriesM10 - Release 0.13David Diederichd.diederich@opengroup.orgAnuj GuptaBhushan RadeShrikant GargDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/209Disables unsafe subst for log4j2023-08-18T11:17:52ZOleksii Tsyganov (EPAM)Disables unsafe subst for log4jM10 - Release 0.13Oleksii Tsyganov (EPAM)Oleksii Tsyganov (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/212Fix log4j (GONRG-4016)2021-12-21T03:36:41ZArtem Dobrynin (EPAM)Fix log4j (GONRG-4016)Updated version to 2.17.0 to fix vulnerabilities
https://community.opengroup.org/groups/osdu/platform/-/epics/11
Part of the #135 seriesUpdated version to 2.17.0 to fix vulnerabilities
https://community.opengroup.org/groups/osdu/platform/-/epics/11
Part of the #135 seriesM10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/213Code coverage2023-08-18T11:17:50ZAalekh JainCode coverageTasks accomplished as part of this MR
1. Added the support for generating code coverage reports using `Jacoco`
2. Updated the settings for CI/CD pipelines (Under section General Pipelines) to capture the code coverage using the Test Cov...Tasks accomplished as part of this MR
1. Added the support for generating code coverage reports using `Jacoco`
2. Updated the settings for CI/CD pipelines (Under section General Pipelines) to capture the code coverage using the Test Coverage Parsing regex as: `Total.*?([0-9]{1,3})%`
3. Added code coverage badge for azure in the `README.md` corresponding to azure cloud provider.
Attached below are the screenshots of the code coverage captured through bade and in the pipeline stages -
1. ![image](/uploads/4a53a8708293ed392a535e2d0186f455/image.png)
2. ![image](/uploads/4e989b4ccad8ab35203cfb634a70b2ed/image.png)M10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/214Upgrading library versions to incorporate the latest Log4j fix2023-08-18T11:17:49ZDavid Diederichd.diederich@opengroup.orgUpgrading library versions to incorporate the latest Log4j fixThese updates apply version 2.17, addressing CVE-2021-45105
Part of the #135 seriesThese updates apply version 2.17, addressing CVE-2021-45105
Part of the #135 seriesM10 - Release 0.13David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/217Azure deploy fixes2023-08-18T11:17:47ZAalekh JainAzure deploy fixesM10 - Release 0.13