Ingestion Workflow merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests2023-08-18T11:18:58Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/122Fixed exclusion at azure file share2023-08-18T11:18:58ZVineeth Guna [Microsoft]Fixed exclusion at azure file shareM7 - Release 0.10https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/126Send Conflict Status When Workflow Already Exists (GONRG-2507)2021-06-10T12:13:33ZAnastasiia GelmutSend Conflict Status When Workflow Already Exists (GONRG-2507)# Description:
Fixed workflow creation process. Sending Conflict Status if workflow already exists.
# How to test:
Ingestion Workflow service can be tested with the help of business cases.
# Changes include:
- [ ] Refactor (a non-break...# Description:
Fixed workflow creation process. Sending Conflict Status if workflow already exists.
# How to test:
Ingestion Workflow service can be tested with the help of business cases.
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [x] Bugfix (a non-breaking change that solves an issue).
- [ ] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
# Dev Checklist:
* [x] Added Unit Tests, wherever applicable.
* [ ] Updated the Readme, if applicable.
* [x] Existing Tests pass
* [x] Verified functionality locally
* [x] Self Reviewed my code for formatting and complex business logic.M7 - Release 0.10Riabokon Stanislav(EPAM)[GCP]Rostislav Dublin (EPAM)Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/125[Azure] Added multi partition support for storage account2021-06-17T09:18:50ZAalekh Jain[Azure] Added multi partition support for storage accountcc: @harshit283 , @kibattul
Related MR in core-lib-azure https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/110/cc: @harshit283 , @kibattul
Related MR in core-lib-azure https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/110/M7 - Release 0.10https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/124Fixed issue in swagger input parameter by upgrading Swagger 2 to Swagger 32021-06-22T12:28:26ZAbhiman NeelakanteswaraFixed issue in swagger input parameter by upgrading Swagger 2 to Swagger 3## Motivation
A bug was raised mentioned that the input parameters in the `params` field for the GET /{workflow_name}/workflowRun API were not accepted. The API did not work from the swagger page but however, worked with Postman.
## C...## Motivation
A bug was raised mentioned that the input parameters in the `params` field for the GET /{workflow_name}/workflowRun API were not accepted. The API did not work from the swagger page but however, worked with Postman.
## Changes
The issue was Swagger 2 was unable to serialize parameters from JSON input to a GET API. This feature is only supported in Open API Spec(OAS)>3.0. Hence, the springfox-swagger version has been updated to 3.0.0
## Known Issues
Recommended library for OAS 3.0 is springdoc-openapi-ui. However, the latest version of this library has an inline script in the index.html of the OAS page. This is currently not allowed according to the security policy in our azure deployment.M7 - Release 0.10Abhiman NeelakanteswaraAbhiman Neelakanteswarahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/134Add AirflowConfigResolver to get airflow configuration for different data par...2023-08-18T11:18:50ZVineeth Guna [Microsoft]Add AirflowConfigResolver to get airflow configuration for different data partitionsM7 - Release 0.10Vineeth Guna [Microsoft]Vineeth Guna [Microsoft]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/135Fixed os core lib azure version as there is a change in version from 0.11.0 t...2023-08-18T11:18:48ZVineeth Guna [Microsoft]Fixed os core lib azure version as there is a change in version from 0.11.0 to 0.10.0 recentlyM7 - Release 0.10https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/133[Azure] Added fileshare support2023-08-18T11:18:52ZAalekh Jain[Azure] Added fileshare supportcc: @vineethgunacc: @vineethgunaM7 - Release 0.10https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/132Ignore DAG content in Create Workflow API2023-08-18T11:18:53ZAbhiman NeelakanteswaraIgnore DAG content in Create Workflow API## Motivation
POST /workflow API used to create new workflows accepts python script under dagContent attribute. As there is no vulnerability scan done on this script presently, it poses a security concern of malicious script injection.
...## Motivation
POST /workflow API used to create new workflows accepts python script under dagContent attribute. As there is no vulnerability scan done on this script presently, it poses a security concern of malicious script injection.
## Changes
To temporarily handle this concern until a more robust solution is in place, the dagContent is being ignored based on `osdu.azure.airflow.ignoreDagContent` value to be set in application.properties of the provider module before deploying. For ease of setting this variable, it has been made an deployment environment variable as `ignore_dagContent`M7 - Release 0.10Abhiman NeelakanteswaraAbhiman Neelakanteswarahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/119[737153:] fix whitesource vulnerabilities2021-07-01T15:07:19ZMaksim Malkov[737153:] fix whitesource vulnerabilities## Motivation
Recently we've conducted a WhiteSource vulnerabilities report review for **non-opersource** version of the Workflow service.
It was suggested to make the same changes for the **open-sourced** version of the service.
So that...## Motivation
Recently we've conducted a WhiteSource vulnerabilities report review for **non-opersource** version of the Workflow service.
It was suggested to make the same changes for the **open-sourced** version of the service.
So that's how this PR was created.
## Changes
Updated dependencies for `Core` and `Azure` modules.
## Issue raised
[issue_120](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/120)
## WhiteSource Reports
You can find report's attached below to this PR.<br>
[workflow-azure-vulnerability-report.xlsx](/uploads/8491838b06a3549876e0ab3da2e399c4/workflow-azure-vulnerability-report.xlsx)
<br>
[workflow-core-vulnerability-report.xlsx](/uploads/752245ca0fd027b014adda1bce155db5/workflow-core-vulnerability-report.xlsx)M7 - Release 0.10Maksim MalkovMaksim Malkovhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/136move workflow name validation to core code2023-08-18T11:18:46ZYunhua Koglinmove workflow name validation to core codemove workflow name validation to core codemove workflow name validation to core codeM7 - Release 0.10Yunhua KoglinYunhua Koglinhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/130Preetisingh/shared dags2023-08-18T11:18:55Zpreeti singh[Microsoft]Preetisingh/shared dagsThis MR creates a new endpoint for creating/deleting SYSTEM Workflows.
It also changes the logic of Getworkflow and GetAllWorkflows to return not only the tenant's workflows but also return the system workflows.This MR creates a new endpoint for creating/deleting SYSTEM Workflows.
It also changes the logic of Getworkflow and GetAllWorkflows to return not only the tenant's workflows but also return the system workflows.M7 - Release 0.10preeti singh[Microsoft]preeti singh[Microsoft]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/137Changed authorize_api to v22023-08-18T11:18:45ZOleksii Tsyganov (EPAM)Changed authorize_api to v2M7 - Release 0.10Oleksii Tsyganov (EPAM)Oleksii Tsyganov (EPAM)