Ingestion Workflow merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests2021-11-29T13:34:53Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/202Add airflow version for info endpoint, enable stable API for gcp (GONRG-3777)2021-11-29T13:34:53ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comAdd airflow version for info endpoint, enable stable API for gcp (GONRG-3777)# Description:
Issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/132 <br/>
Add a version of Airflow into an endpoint 'info' for Workflow Service.<br/>
A version of Airflow will be reques...# Description:
Issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/132 <br/>
Add a version of Airflow into an endpoint 'info' for Workflow Service.<br/>
A version of Airflow will be requested only when enabled stable API `osdu.airflow.version2=true`.<br/>
Enable stable API for gcp.
# How to test:
Functionality can be testes with a request:
~~~
curl --location --request GET 'https://com.osdu.club/api/workflow/info' \
~~~
Response example:
~~~
{
"groupId": "org.opengroup.osdu",
"artifactId": "workflow-gcp",
"version": "0.13.0-SNAPSHOT",
"buildTime": "2021-11-25T12:29:41.581Z",
"branch": "add-airflow-version-for-info-endpoint",
"commitId": "5e0816ea1088e6205fae839db389ebf403dc7885",
"commitMessage": "Add airflow version for info endpoint, enable stable API for gcp",
"connectedOuterServices": [
{
"name": "Airflow",
"version": "2.1.2+composer"
}
]
}
~~~
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [x] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
- [x] Common code
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/204Add a version of Airflow into an endpoint 'info' for Workflow Service(GONRG-3...2021-12-15T19:43:19ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comAdd a version of Airflow into an endpoint 'info' for Workflow Service(GONRG-3777)# Description:
Issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/132 <br/>
Add a version of Airflow into an endpoint 'info' for Workflow Service.<br/>
A version of Airflow will be reques...# Description:
Issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/132 <br/>
Add a version of Airflow into an endpoint 'info' for Workflow Service.<br/>
A version of Airflow will be requested only when enabled stable API `osdu.airflow.version2=true`.<br/>
# How to test:
Functionality can be testes with a request:
~~~
curl --location --request GET 'https://com.osdu.club/api/workflow/info' \
~~~
Response example:
~~~
{
"groupId": "org.opengroup.osdu",
"artifactId": "workflow-gcp",
"version": "0.13.0-SNAPSHOT",
"buildTime": "2021-11-25T12:29:41.581Z",
"branch": "add-airflow-version-for-info-endpoint",
"commitId": "5e0816ea1088e6205fae839db389ebf403dc7885",
"commitMessage": "Add airflow version for info endpoint, enable stable API for gcp",
"connectedOuterServices": [
{
"name": "Airflow",
"version": "2.1.2+composer"
}
]
}
~~~
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [x] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [ ] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
- [x] Common code
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/191Added iaap auth for airflow stable api (GONRG-3753)2021-11-19T11:36:12ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comAdded iaap auth for airflow stable api (GONRG-3753)# Description:
Refactor workflow to use different approaches for auth with Airflow 2.0.
This [MR](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/160) brings Airflow 2.0 stable API to...# Description:
Refactor workflow to use different approaches for auth with Airflow 2.0.
This [MR](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/160) brings Airflow 2.0 stable API to workflow service, but it has hardcoded Basic auth as the only way to authenticate requests to Airflow. We need to un-hardcode authentication and verify work with Airflow 2.0 stable API.
Airflow2 can be deployed either in Composer or K8S.
If it is deployed in Composer, then Authorization remains the same as it was before (Bearer: <IAAP_Token>), because Composer uses its own auth backend for Stable API
If Airflow is deployed in K8S, then we can use Basic Auth.
Workflow Service must have the possibility to choose one of these two options depending on where AF2 is deployed.
In order to switch between `stable` and `experimental` API env variable `OSDU_AIRFLOW_VERSION2` may be used.<br/>
In order to switch between `basic` and `IAAP` auth env variable `AIRFLOW_IAAP_MODE` may be used.
# How to test:
Functionality was tested with `experimental` and `stable` API
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [x] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
- [ ] Common code
# Dev Checklist:
- [x] Added Unit Tests, wherever applicable.
- [x] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Rostislav Dublin (EPAM)Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/194Airflow2 stable api support2023-08-18T11:18:04ZSaurabh TripathiAirflow2 stable api support### Summary of changes
* Make changes to the connect to airflow2 stable API and accommodate the response changes
* UT for the airflow2 implementations
### Related changes
* add airflow env selection variable in pipeline
* Documentation### Summary of changes
* Make changes to the connect to airflow2 stable API and accommodate the response changes
* UT for the airflow2 implementations
### Related changes
* add airflow env selection variable in pipeline
* DocumentationM10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/219Airflow2 upgrade doc2023-08-18T11:17:45ZSaurabh TripathiAirflow2 upgrade docM10 - Release 0.13Vineeth Guna [Microsoft]Vineeth Guna [Microsoft]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/197[Azure] Add support for caching in workflow ingestion service2023-08-18T11:18:02ZAalekh Jain[Azure] Add support for caching in workflow ingestion serviceM10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/217Azure deploy fixes2023-08-18T11:17:47ZAalekh JainAzure deploy fixesM10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/199[Azure] Limiting trigger workflow requests in workflow ingestion service2023-08-18T11:18:00ZAalekh Jain[Azure] Limiting trigger workflow requests in workflow ingestion serviceM10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/236Cherry-pick 'fix json-smart version' into 'release/0.13'2022-02-04T19:53:36ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'fix json-smart version' into 'release/0.13'Original MR: !235Original MR: !235M10 - Release 0.13David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/213Code coverage2023-08-18T11:17:50ZAalekh JainCode coverageTasks accomplished as part of this MR
1. Added the support for generating code coverage reports using `Jacoco`
2. Updated the settings for CI/CD pipelines (Under section General Pipelines) to capture the code coverage using the Test Cov...Tasks accomplished as part of this MR
1. Added the support for generating code coverage reports using `Jacoco`
2. Updated the settings for CI/CD pipelines (Under section General Pipelines) to capture the code coverage using the Test Coverage Parsing regex as: `Total.*?([0-9]{1,3})%`
3. Added code coverage badge for azure in the `README.md` corresponding to azure cloud provider.
Attached below are the screenshots of the code coverage captured through bade and in the pipeline stages -
1. ![image](/uploads/4a53a8708293ed392a535e2d0186f455/image.png)
2. ![image](/uploads/4e989b4ccad8ab35203cfb634a70b2ed/image.png)M10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/165Disable accepting dag content based on feature flag [azure]2023-08-18T11:18:27ZAalekh JainDisable accepting dag content based on feature flag [azure]M10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/201Disable spring web security 2052023-08-18T11:17:57ZBhushan RadeDisable spring web security 205M10 - Release 0.13Anuj GuptaBhushan RadeAnuj Guptahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/209Disables unsafe subst for log4j2023-08-18T11:17:52ZOleksii Tsyganov (EPAM)Disables unsafe subst for log4jM10 - Release 0.13Oleksii Tsyganov (EPAM)Oleksii Tsyganov (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/203Enable airflow stable api for gcp2021-11-29T15:31:55ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comEnable airflow stable api for gcp# Description:
Enable stable API for gcp. Minor refactoring & unhardcode integration tests.
~~~
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solv...# Description:
Enable stable API for gcp. Minor refactoring & unhardcode integration tests.
~~~
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [x] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/212Fix log4j (GONRG-4016)2021-12-21T03:36:41ZArtem Dobrynin (EPAM)Fix log4j (GONRG-4016)Updated version to 2.17.0 to fix vulnerabilities
https://community.opengroup.org/groups/osdu/platform/-/epics/11
Part of the #135 seriesUpdated version to 2.17.0 to fix vulnerabilities
https://community.opengroup.org/groups/osdu/platform/-/epics/11
Part of the #135 seriesM10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/221GCP Fix for request with 100k manifest entries (GONRG-4050)2022-01-10T14:12:51ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comGCP Fix for request with 100k manifest entries (GONRG-4050)# Description:
Change airflow request building to reduce heap occupation.
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [x] Bugfix (a non-breaking change that solves an issue).
- [ ] Ne...# Description:
Change airflow request building to reduce heap occupation.
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [x] Bugfix (a non-breaking change that solves an issue).
- [ ] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/182Gcp workflow osm(GONRG-3404)2021-11-03T09:23:18ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comGcp workflow osm(GONRG-3404)# Description:
Implementation of new common code design based on OSM framework.
# How to test:
Functionality was tested with Datastore and Postgres integration tests passed.
# Changes include:
- [x] Refactor (a non-breaking change that...# Description:
Implementation of new common code design based on OSM framework.
# How to test:
Functionality was tested with Datastore and Postgres integration tests passed.
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [x] New feature (a non-breaking change that adds functionality).
# Changes in:
- [x] GCP
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [x] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.
# Other comments:
Migration cannot be done in one step, wanting to keep the current data structure (not model native),
this MR includes several classes that can be removed later:
~~~
`@ConditionalOnProperty(name = "datastore.legacy.data.structure", havingValue = "true")`
~~~
OSM framework is currently **not published** to the community, and the package registry does not have an artifact of it, the build **will fail** until OSM is published.M10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Rostislav Dublin (EPAM)Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/226GONRG-4118 Add imagePullPolicy value2023-08-18T11:17:37ZIgor Zimovets (EPAM)GONRG-4118 Add imagePullPolicy valueM10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/227GONRG-4206: Verify dev2 pipeline for workflow2023-08-18T11:17:36ZNastassia Rabeichykava (EPAM)GONRG-4206: Verify dev2 pipeline for workflowM10 - Release 0.13Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/220Ibm core lib upgrade2023-08-18T11:17:44ZAshwani PandeyIbm core lib upgradeIBM core lib version upgrade to rc6IBM core lib version upgrade to rc6M10 - Release 0.13Ashwani PandeyAshwani Pandey