Data Ingestion issueshttps://community.opengroup.org/groups/osdu/platform/data-flow/ingestion/-/issues2023-07-05T10:09:41Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/73ADR: Workflow Service - Support for custom operators2023-07-05T10:09:41ZKishore BattulaADR: Workflow Service - Support for custom operators# Context
Composability and re-usability are important aspects when designing workflows. This ADR covers APIs needed to achieve composability using re-usable components using airflow as a workflow engine.
- All the necessary python libr...# Context
Composability and re-usability are important aspects when designing workflows. This ADR covers APIs needed to achieve composability using re-usable components using airflow as a workflow engine.
- All the necessary python libraries for operators and DAGs (Directed Acyclic Graphs) are pre-installed in Airflow.
One of the mechanisms to achieve composability and re-usability is through airflow custom operators. API will be provided with a workflow service to register a custom operator in airflow. This custom operator is one of the building blocks for composability and re-usability. Once the operator is registered, it can be used in a DAG (Directed Acyclic Graph).
# Scope
API specification can be found [here](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/blob/custom-operators-spec/docs/api/openapi.yaml)
![image](/uploads/6bedf852fa6fe47b36782d63d4e79b96/image.png)
# Out of Scope
- Support for custom libraries at run time is not supported.
- Validation of Operators is not supported.
- Update and delete operators are not supported.
- Custom Hooks, Sensor are not supported.
- Packaged DAGs with DAGs, Custom Operators is not supported.
# Decision
- The decision is to add the above-mentioned APIs to the workflow service.
- The registration of custom operators is expected to be aligned with Airflow recommended practices
# Rationale
These APIs will provide a customer to register custom operators at run time and can be used in multiple DAGs. This helps to re-use operators across DAGs
# Consequences
CSPs need to implement these new APIs to support custom operators.
# High level Architecture
![image](/uploads/d6ced893addfc249d77434f9b851fb4b/image.png)
## Intermediate Data Sharing
One of the key components in running workflows is intermediate sharing data between different tasks. For example, when ingesting 100 MB csv file, each step in the workflow needs to pass the transformed version of this file to next step. Airflow xcom can be used to share metadata between different tasks but it is not sufficient for sharing large files. Airflow xcom has limitation on the amount of data that can be share. To facilitate this a new API is provided on workflow service which will return a signedURL. Using this signed URL multiple files can be created and information about these files can be shared using airflow xcom. Below steps should provide a overview of how this signedURL functionality is going to be used.
- Every workflow will request a signed URL at the start of the workflow.
- Each task will use the same signed url to save intermediate data.
- Tasks will communicate the information about intermediate data through xcom.
## Examples
### Sample Custom Operator
```
from airflow.operators.python_operator import PythonOperator
from airflow.exceptions import AirflowException
class PrintRecord(PythonOperator):
def __init__(self, *args, **kwargs) -> None:
super().__init__(python_callable = self.my_callable,*args, **kwargs)
self.args = args
self.kwargs = kwargs
def my_callable(self, **kwargs):
print("Printing data")
ti = kwargs['ti']
pulled_value = ti.xcom_pull(task_ids='get_record')
print(pulled_value)
```
### Sample DAG Using Custom Operator and Http operator
```
from airflow import DAG
from airflow.operators import BashOperator
from datetime import datetime, timedelta
from airflow.utils.dates import days_ago
from operators.print_record_operator import PrintRecord
from airflow.operators.http_operator import SimpleHttpOperator
from airflow.operators.python_operator import PythonOperator
token = "{{ dag_run.conf['authToken'] }}"
data_partition_id = "{{ dag_run.conf['runConfiguration']['dataPartitionId'] }}"
kind = "{{ dag_run.conf['runConfiguration']['kind'] }}"
record_id = "{{ dag_run.conf['runConfiguration']['id'] }}"
storage_host = "{{ dag_run.conf['additionalProperties']['storageHost'] }}"
storage_endpoint = "api/storage/v2/records/" + record_id
# Following are defaults which can be overridden later on
default_args = {
'start_date': days_ago(0),
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=1),
}
dag = DAG('Print_Storage_Record', default_args=default_args)
t1 = SimpleHttpOperator(
http_conn_id='http_storage',
task_id='get_record',
endpoint=storage_endpoint,
method='get',
headers={"Authorization":token,"data-partition-id":"opendes","Accept":"application/json"},
xcom_push=True,
log_response=True,
dag=dag
)
t2 = PrintRecord(task_id='custom-task', provide_context=True, dag=dag)
t1 >> t2
```
### CSV DAG built using custom operator.
![image](/uploads/cdb536e671528f9edd9a6c02f99d540f/image.png)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/116Airflow Experimental Delete API not implemented for RBAC enabled Airflow.2021-06-14T16:39:57ZMayank Saggar [Microsoft]Airflow Experimental Delete API not implemented for RBAC enabled Airflow.There is a issue with Airflow where if RBAC for Webserver is enabled, the delete API from airflow was returning 404. On digging the airflow source code, it was discovered that there is separate handler for api requests with rbac enabled ...There is a issue with Airflow where if RBAC for Webserver is enabled, the delete API from airflow was returning 404. On digging the airflow source code, it was discovered that there is separate handler for api requests with rbac enabled which doesn't provide delete API.
Github Source: [endpoints when rbac enabled](https://github.com/apache/airflow/blob/1.10.12/airflow/www_rbac/api/experimental/endpoints.py)
[endpoints when rbac disabled](https://github.com/apache/airflow/blob/1.10.12/airflow/www/api/experimental/endpoints.py)
So if we remove RBAC from webserver we get the Experimental delete API.https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/152APIs to get the XCOM summary (Entries) are working in AWS environment, but ar...2023-11-09T07:43:12ZKamlesh TodaiAPIs to get the XCOM summary (Entries) are working in AWS environment, but are NOT working in other CSPs (Azure, GC and IBM) environmentsThe APIs to get the xcomEntries using the the runid and the taskinstance are working in AWS environment. The endpoints/API are not implemented/deployed in other CSP's (Azure, GC, IBM) environments.
<details><summary>curl --location 'htt...The APIs to get the xcomEntries using the the runid and the taskinstance are working in AWS environment. The endpoints/API are not implemented/deployed in other CSP's (Azure, GC, IBM) environments.
<details><summary>curl --location 'https://r3m16.forumtesting.osdu.aws/api/airflow/api/v1/dags/Osdu_ingest/dagRuns/45eb9f45-aada-4e2c-b618-818fb5dfcf28/taskInstances/process_single_manifest_file_task/**xcomEntries/record_ids**' \
--header 'data-partition-id: osdu' \
--header 'Authorization: Bearer eyJraWQiOi...fWbOUA3RcQ'</summary>
</details>
Response 200 OK
{
"dag_id": "Osdu_ingest",
"execution_date": "2023-04-04T21:19:27.327451+00:00",
"key": "record_ids",
"task_id": "process_single_manifest_file_task",
"timestamp": "2023-04-04T21:19:48.761929+00:00",
"value": "['osdu:reference-data--FacilityType:WELL_999259423605', 'osdu:master-data--Organisation:Auto_Test_999259423605', 'osdu:reference-data--FacilityEventType:SPUD_DATE_999259423605', 'osdu:reference-data--VerticalMeasurementPath:DEPTH_DATUM_ELEV_999259423605', 'osdu:reference-data--AliasNameType:WELL_NAME_999259423605', 'osdu:master-data--Well:999259423605']"
}
=========================================================================================================================
`curl --location 'https://r3m16.forumtesting.osdu.aws/api/airflow/api/v1/dags/Osdu_ingest/dagRuns/45eb9f45-aada-4e2c-b618-818fb5dfcf28/taskInstances/process_single_manifest_file_task/**xcomEntries/skipped_ids**' \
--header 'data-partition-id: osdu' \
--header 'Authorization: Bearer eyJraWQiOi...fWbOUA3RcQ`
Response 200 OK
{
"dag_id": "Osdu_ingest",
"execution_date": "2023-04-04T21:19:27.327451+00:00",
"key": "skipped_ids",
"task_id": "process_single_manifest_file_task",
"timestamp": "2023-04-04T21:19:48.783236+00:00",
"value": "[]"
}
@chad @debasisc @Srinivasan_Narayanan @dzmitry_malkevich @anujguptahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/40Azure | Core - Intermittent Failures while calling schema service API2022-06-27T21:34:42Zharshit aggarwalAzure | Core - Intermittent Failures while calling schema service APISample Error - I/O error on PUT request for "https://<fqdn>/api/schema-service/v1/schema": Unexpected end of file from server; nested exception is java.net.SocketException: Unexpected end of file from server
Pipeline runs for reference
...Sample Error - I/O error on PUT request for "https://<fqdn>/api/schema-service/v1/schema": Unexpected end of file from server; nested exception is java.net.SocketException: Unexpected end of file from server
Pipeline runs for reference
https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/jobs/468050
https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/jobs/467621https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/41Azure | Intermittent failures for Azure Integration Tests2022-06-27T21:30:44Zharshit aggarwalAzure | Intermittent failures for Azure Integration TestsWe have been observing random failures in Azure Integration Tests for CSV Parser. As the integration tests for CSV Parser involves searching and indexing of records hence any delays during those can cause failures here, to overcome this ...We have been observing random failures in Azure Integration Tests for CSV Parser. As the integration tests for CSV Parser involves searching and indexing of records hence any delays during those can cause failures here, to overcome this we need to have more robust and resillient retry mechanisms
This [MR](https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/119) has been merged despite failures in [Azure Tests](https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/jobs/471776) hence fix for this should be contributed at the earliestSwapnilAbhishek KumarSwapnilhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/26Cloud Datasource Support - MSFT Azure2022-07-27T05:48:32ZKateryna Kurach (EPAM)Cloud Datasource Support - MSFT AzurePre-condition:
OSDU instance is deployed to MSFT Azure.
Cloud Storage on this diagram is a storage location outside the OSDU instance. It is located in the same MSFT Azure Cloud as OSDU instance. MSFT Team has to create a connector for ...Pre-condition:
OSDU instance is deployed to MSFT Azure.
Cloud Storage on this diagram is a storage location outside the OSDU instance. It is located in the same MSFT Azure Cloud as OSDU instance. MSFT Team has to create a connector for the Ingestion framework that enables the transfer and ingestion of raw files from the Cloud Storage into an OSDU instance. It’s up to MSFT team to decide what types of Azure Data Services should be supported in R3 scope (Azure Files, Azure Tables etc).
Acceptance Criteria:
1. Files are put into Azure Datasource
2. It is possible to configure an Ingestion workflow that will source files from the Cloud Datasource and store them in the OSDU File Storage.Dania Kodeih (Microsoft)Dania Kodeih (Microsoft)https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/eds-dms/-/issues/3Confirm that EdsDmsApi is implementing the common interface2023-02-13T15:19:14ZRene von Borstel [EPAM]Confirm that EdsDmsApi is implementing the common interfacePlease confirm that this Class is implementing the common interface at
https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/dms/IDmsService.java
[Referen...Please confirm that this Class is implementing the common interface at
https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/dms/IDmsService.java
[Reference](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/eds-dms/-/merge_requests/2#note_133928)Rene von Borstel [EPAM]Rene von Borstel [EPAM]https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser-v2/-/issues/13M20/Azure/Preship - failure handling Trajectory data2023-10-04T02:22:40ZDebasis ChatterjeeM20/Azure/Preship - failure handling Trajectory dataUsed this source data[Trajectory-WITSML-DC.txt](/uploads/9732cc2a456502e990593b58b641229d/Trajectory-WITSML-DC.txt)
Dataset record opendes:dataset--File.Generic:b5c31b0cacdf4238b13f098beb382694
Energyml_converter runID="runId": "f2051ce...Used this source data[Trajectory-WITSML-DC.txt](/uploads/9732cc2a456502e990593b58b641229d/Trajectory-WITSML-DC.txt)
Dataset record opendes:dataset--File.Generic:b5c31b0cacdf4238b13f098beb382694
Energyml_converter runID="runId": "f2051ce6-5bfc-4374-af89-0b2582fb4b9b",
Xcom summary shows record ID =
{'record_id': 'opendes:dataset--File.Generic:5d75967ee5384e6e8f7f5f38cb78ec3b:',
Using the above dataset record, we run Manifest Ingestion by reference.
"runId": "223ec4fc-b0c5-48df-bfaa-f4fe71047ed2",
Fails
Step provide_manifest_integrity_task
[{'id': 'opendes:work-product--WorkProduct:b102ceb0-41f2-452a-8fbf-762c5497d2d7', 'kind': 'osdu:wks:work-product--WorkProduct:1.0.0', 'reason': 'Missing parents: set()'}]
See JSON file created by EnergyML_converter[Generates-manifest.json](/uploads/76464c6f8b61446cffcd42c83f466e79/Generates-manifest.json)
There are many problems with the use of trailing colon.Valentin GauthierValentin Gauthierhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser-v2/-/issues/11M20/Azure/Preship - Wellbore data type - ID created with trailing colon2023-10-03T08:44:02ZDebasis ChatterjeeM20/Azure/Preship - Wellbore data type - ID created with trailing colonIt is not common convention for ID to have trailing colon.
Please consider revision to remove this.
['opendes:work-product--WorkProduct:fc6be8ae-83f0-4394-a164-7d8786595522',
'opendes:dataset--File.WITSML:411dddd1c4bf430ca45ee31ac155e9...It is not common convention for ID to have trailing colon.
Please consider revision to remove this.
['opendes:work-product--WorkProduct:fc6be8ae-83f0-4394-a164-7d8786595522',
'opendes:dataset--File.WITSML:411dddd1c4bf430ca45ee31ac155e9e6',
'opendes:master-data--Wellbore:3c321e10-d04a-4b6d-8bcd-5eae01cab52d:']
Note Wellbore record created with trailing colon in ID.Valentin GauthierValentin Gauthierhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser-v2/-/issues/10M20/Azure/Preship - Wellbore XML file - job fails with no meaningful error me...2023-09-29T07:22:21ZDebasis ChatterjeeM20/Azure/Preship - Wellbore XML file - job fails with no meaningful error message[Wellbore21-Peter-DC.xml](/uploads/ba28d197eb9a8604d3d7de934fb70a1a/Wellbore21-Peter-DC.xml)
Provided by @pgonzalez71 .
Tried to run the workflow using this data file.
Dataset record ID = opendes:dataset--File.Generic:315d6f9b8f7f4edeb...[Wellbore21-Peter-DC.xml](/uploads/ba28d197eb9a8604d3d7de934fb70a1a/Wellbore21-Peter-DC.xml)
Provided by @pgonzalez71 .
Tried to run the workflow using this data file.
Dataset record ID = opendes:dataset--File.Generic:315d6f9b8f7f4edebb029ace3a216f99
runID = "runId": "d42c3ad3-9b4e-458a-aba5-a4daaf08a34a",
Airflow log from step of content_loading.
[Airflow-log-content-loading.txt](/uploads/21c53290e8821686acee4e15dc3a4067/Airflow-log-content-loading.txt)Valentin GauthierValentin Gauthierhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/150Misleading log statements2022-12-12T15:35:32ZMaksim MalkovMisleading log statementsWorkflow service search for a triggered workflow first in provided data partition. System workflow like CSV would not be available in data partition. In such cases service publish logs "workflow not found"
Next same workflow is searched ...Workflow service search for a triggered workflow first in provided data partition. System workflow like CSV would not be available in data partition. In such cases service publish logs "workflow not found"
Next same workflow is searched in system db and it is found there and processing completes
But these logs are creating a confusion that some workflow is not found by workflow service, but actually there is no such issue.M16 - Release 0.19https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/issues/112Osdu_Ingest - fails to detect issue with GeoJSON structure in Spatial block2024-01-22T15:10:15ZDebasis ChatterjeeOsdu_Ingest - fails to detect issue with GeoJSON structure in Spatial blockEnvironment M22/Azure/Preship
Osdu_Ingest run ID = aaf9eaa0-beec-421f-8cbe-f87f6373ed4e
['opendes:master-data--SeismicAcquisitionSurvey:DC19JAN']
Input JSON payload has some issue with GeoJSON part.
validate_manifest_schema_task show...Environment M22/Azure/Preship
Osdu_Ingest run ID = aaf9eaa0-beec-421f-8cbe-f87f6373ed4e
['opendes:master-data--SeismicAcquisitionSurvey:DC19JAN']
Input JSON payload has some issue with GeoJSON part.
validate_manifest_schema_task shows clean log.
Later the record gets created.
When we check record using Storage, it shows Spatial data (WGS84 coordinates).
Search does not show that information any more.
Troubleshooting reveals that Spatial data is not indexed since GeoJSON syntax is incorrect.
"geo-json shape parsing error: must be a valid FeatureCollection attribute: SpatialLocation.Wgs84Coordinates",
The question - why did "schema validation" step not detect the problem and stop record creation?
cc @Yan_Sushchynski
See enclosed file with additional information.
[2024-01-19-EDS-Seismic.txt](/uploads/4175777071cda54bd465b41eb0264f30/2024-01-19-EDS-Seismic.txt)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/114POST /v1/workflow/{workflow_name}/workflowRun (Trigger Workflow) with empty r...2021-06-14T16:18:30ZAalekh JainPOST /v1/workflow/{workflow_name}/workflowRun (Trigger Workflow) with empty runId throws 500 internal server error## Description
**Current Behaviour**
For the given request body
```json
{
"runId": "",
"executionContext": {
}
}
```
Error thrown is
```json
{
"code": 500,
"reason": "Unexpectedly failed to insert item into CosmosDB",
...## Description
**Current Behaviour**
For the given request body
```json
{
"runId": "",
"executionContext": {
}
}
```
Error thrown is
```json
{
"code": 500,
"reason": "Unexpectedly failed to insert item into CosmosDB",
"message": "[\"The input name '' is invalid. Ensure to provide a unique non-empty string less than '1024' characters.\"], {\"userAgent\":\"azsdk-java-cosmos/4.7.1 Windows10/10.0 JRE/1.8.0_265\",\"requestLatencyInMs\":212,\"requestStartTimeUTC\":\"2021-04-19T09:01:33.929Z\",\"requestEndTimeUTC\":\"2021-04-19T09:01:34.141Z\",\"connectionMode\":\"DIRECT\",\"responseStatisticsList\":[{\"storeResult\":{\"storePhysicalAddress\":\"rntbd://cdb-ms-prod-eastus2-fd7.documents.azure.com:14178/apps/a78846d5-27aa-45e8-bef0-0950c8a3c1d2/services/9a98fb60-a0fb-43e4-be05-3fe8dc8d6498/partitions/e158dec0-4caf-42e6-b7e8-1eb9dc9b7c84/replicas/132593652864958972p/\",\"lsn\":84405,\"globalCommittedLsn\":84405,\"partitionKeyRangeId\":\"1\",\"isValid\":true,\"statusCode\":400,\"subStatusCode\":0,\"isGone\":false,\"isNotFound\":false,\"isInvalidPartition\":false,\"requestCharge\":1.24,\"itemLSN\":-1,\"sessionToken\":\"-1#84405\",\"exception\":\"[\\\"The input name '' is invalid. Ensure to provide a unique non-empty string less than '1024' characters.\\\"]\",\"transportRequestTimeline\":[{\"eventName\":\"created\",\"durationInMicroSec\":\"0\",\"startTime\":\"2021-04-19T09:01:33.931Z\"},{\"eventName\":\"queued\",\"durationInMicroSec\":\"0\",\"startTime\":\"2021-04-19T09:01:33.931Z\"},{\"eventName\":\"channelAcquisitionStarted\",\"durationInMicroSec\":\"3000\",\"startTime\":\"2021-04-19T09:01:33.931Z\"},{\"eventName\":\"pipelined\",\"durationInMicroSec\":\"1000\",\"startTime\":\"2021-04-19T09:01:33.934Z\"},{\"eventName\":\"transitTime\",\"durationInMicroSec\":\"204000\",\"startTime\":\"2021-04-19T09:01:33.935Z\"},{\"eventName\":\"received\",\"durationInMicroSec\":\"1000\",\"startTime\":\"2021-04-19T09:01:34.139Z\"},{\"eventName\":\"completed\",\"durationInMicroSec\":\"1000\",\"startTime\":\"2021-04-19T09:01:34.140Z\"}],\"rntbdRequestLengthInBytes\":714,\"rntbdResponseLengthInBytes\":325,\"requestPayloadLengthInBytes\":282,\"responsePayloadLengthInBytes\":null,\"channelTaskQueueSize\":1,\"pendingRequestsCount\":1,\"serviceEndpointStatistics\":{\"availableChannels\":1,\"acquiredChannels\":0,\"executorTaskQueueSize\":0,\"inflightRequests\":1,\"lastSuccessfulRequestTime\":\"2021-04-19T08:52:54.424Z\",\"lastRequestTime\":\"2021-04-19T08:52:54.211Z\",\"createdTime\":\"2021-04-19T08:34:55.741Z\",\"isClosed\":false}},\"requestResponseTimeUTC\":\"2021-04-19T09:01:34.141Z\",\"requestResourceType\":\"Document\",\"requestOperationType\":\"Create\"}],\"supplementalResponseStatisticsList\":[],\"addressResolutionStatistics\":{},\"regionsContacted\":[\"https://osdu-mvp-dp1dev-qs29-db-eastus2.documents.azure.com:443/\"],\"retryContext\":{\"retryCount\":0,\"statusAndSubStatusCodes\":null,\"retryLatency\":0},\"metadataDiagnosticsContext\":{\"metadataDiagnosticList\":null},\"serializationDiagnosticsContext\":{\"serializationDiagnosticsList\":[{\"serializationType\":\"ITEM_SERIALIZATION\",\"startTimeUTC\":\"2021-04-19T09:01:33.929Z\",\"endTimeUTC\":\"2021-04-19T09:01:33.929Z\",\"durationInMicroSec\":0}]},\"gatewayStatistics\":null,\"systemInformation\":{\"usedMemory\":\"237293 KB\",\"availableMemory\":\"3432723 KB\",\"systemCpuLoad\":\"(2021-04-19T09:01:08.919Z 5.1%), (2021-04-19T09:01:13.920Z 6.9%), (2021-04-19T09:01:18.921Z 4.0%), (2021-04-19T09:01:23.919Z 5.1%), (2021-04-19T09:01:28.920Z 5.3%), (2021-04-19T09:01:33.922Z 10.5%)\"},\"clientCfgs\":{\"id\":0,\"numberOfClients\":1,\"connCfg\":{\"rntbd\":\"(cto:PT5S, rto:PT5S, icto:PT0S, ieto:PT1H, mcpe:130, mrpc:30)\",\"gw\":\"(cps:1000, rto:PT5S, icto:null, p:false)\",\"other\":\"(ed: true, cs: false)\"},\"consistencyCfg\":\"(consistency: null, mm: true, prgns: [])\"}}"
}
```
**Expected Behaviour**
Should throw an error saying `runId` cannot be empty or invalid `runId` given OR `runId` needs to be generated (similar to what happens when `runId` field is not present in the request body) - Need confirmation on the expected behaviour
Works fine when the request body does not have `runId` as the key
```json
{
"executionContext": {
}
}
```
cc: @kibattul @vineethgunahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/100PUT v1/workflow/{workflow_name}/workflowRun/{runId} doesnt update the status ...2021-05-25T04:51:57ZMonalisa SrivastavaPUT v1/workflow/{workflow_name}/workflowRun/{runId} doesnt update the status of the workflow on the Airflow UIPUT v1/workflow/{workflow_name}/workflowRun/{runId} doesnt update the status of the workflow,
When we try to update the status for a workflow ,though we get a 200 Ok and success response the Airflow still shows the same status.
Howeve...PUT v1/workflow/{workflow_name}/workflowRun/{runId} doesnt update the status of the workflow,
When we try to update the status for a workflow ,though we get a 200 Ok and success response the Airflow still shows the same status.
However from Airflow UI, when we update the status, the status changes suucessfully.Monalisa SrivastavaMonalisa Srivastavahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/eds-dms/-/issues/18Side effect to ingest configuration files of EDS DMS2023-11-23T13:06:23ZRiabokon Stanislav(EPAM)[GCP]Side effect to ingest configuration files of EDS DMSThe GC Team has identified an issue. According to the architectural design of this service, the procedure involves creating configuration files within the Storage Service. Subsequently, new records will be indexed by the Indexer Search a...The GC Team has identified an issue. According to the architectural design of this service, the procedure involves creating configuration files within the Storage Service. Subsequently, new records will be indexed by the Indexer Search and placed into Elastic Search. As a result, these records become discoverable through the Search Service.
Current Arch:
![image](/uploads/82a2a551574d5772a2105b64fcf27950/image.png)
`https://community.gcp.gnrg-osdu.projects.epam.com/api/search/v2/query`
bode:
```
{
"kind": "osdu:wks:reference-data--SecuritySchemeType:1.0.0"
}
```
response:
```
{
"results": [
{
"data": {
"AttributionPublication": null,
"InactiveIndicator": null,
"Description": "An open and industry-standard protocol for authorization",
"ResourceLifecycleStatus": null,
"ResourceCurationStatus": null,
"TechnicalAssuranceID": null,
"Code": "OAuth2",
"Source": "SecuritySchemeType.1.0.0.xlsx",
"Name": "OAuth 2.0",
"AttributionAuthority": "OSDU",
"ResourceHomeRegionID": null,
"VirtualProperties.DefaultName": "OAuth 2.0",
"AttributionRevision": null,
"ResourceSecurityClassification": null,
"ID": "OAuth2",
"ExistenceKind": null
},
"kind": "osdu:wks:reference-data--SecuritySchemeType:1.0.0",
"source": "wks",
"acl": {
"viewers": [
"data.default.viewers@osdu.group"
],
"owners": [
"data.default.owners@osdu.group"
]
},
"type": "reference-data--SecuritySchemeType",
"version": 1697963580525660,
"tags": {
"normalizedKind": "osdu:wks:reference-data--SecuritySchemeType:1"
},
"modifyUser": "osdu-community-sa-airflow@nice-etching-277309.iam.gserviceaccount.com",
"modifyTime": "2023-10-22T08:33:00.665Z",
"createTime": "2022-09-30T10:26:21.248Z",
"authority": "osdu",
"namespace": "osdu:wks",
"legal": {
"legaltags": [
"osdu-demo-legaltag"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"createUser": "osdu-community-sa-airflow@nice-etching-277309.iam.gserviceaccount.com",
"id": "osdu:reference-data--SecuritySchemeType:OAuth2"
},
{
"data": {
"AttributionPublication": null,
"InactiveIndicator": null,
"Description": "Requests are authenticated using an access key, such as a JSON Web Token, in the request header.",
"ResourceLifecycleStatus": null,
"ResourceCurationStatus": null,
"TechnicalAssuranceID": null,
"Code": "Bearer",
"Source": "SecuritySchemeType.1.0.0.xlsx",
"Name": "Bearer Token",
"AttributionAuthority": "OSDU",
"ResourceHomeRegionID": null,
"VirtualProperties.DefaultName": "Bearer Token",
"AttributionRevision": null,
"ResourceSecurityClassification": null,
"ID": "Bearer",
"ExistenceKind": null
},
"kind": "osdu:wks:reference-data--SecuritySchemeType:1.0.0",
"source": "wks",
"acl": {
"viewers": [
"data.default.viewers@osdu.group"
],
"owners": [
"data.default.owners@osdu.group"
]
},
"type": "reference-data--SecuritySchemeType",
"version": 1697963580525660,
"tags": {
"normalizedKind": "osdu:wks:reference-data--SecuritySchemeType:1"
},
"modifyUser": "osdu-community-sa-airflow@nice-etching-277309.iam.gserviceaccount.com",
"modifyTime": "2023-10-22T08:33:00.665Z",
"createTime": "2022-09-30T10:28:21.843Z",
"authority": "osdu",
"namespace": "osdu:wks",
"legal": {
"legaltags": [
"osdu-demo-legaltag"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"createUser": "osdu-community-sa-airflow@nice-etching-277309.iam.gserviceaccount.com",
"id": "osdu:reference-data--SecuritySchemeType:Bearer"
}
],
"aggregations": null,
"totalCount": 2
}
```
It appears there may be a potential security concern within the EDS Service architecture.https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/112Swagger is not behaving correctly for the API /v1/workflow/{workflow_name}/wo...2021-04-07T12:08:26ZMonalisa SrivastavaSwagger is not behaving correctly for the API /v1/workflow/{workflow_name}/workflowRun getAllRunInstancesSwagger https://osdu-glab.msft-osdu-test.org/api/workflow/swagger-ui.html#/workflow-run-api/getAllRunInstancesUsingGET is not accepting params however through postman the API works fine and accept the 4 params `prefix`, `startDate`, `end...Swagger https://osdu-glab.msft-osdu-test.org/api/workflow/swagger-ui.html#/workflow-run-api/getAllRunInstancesUsingGET is not accepting params however through postman the API works fine and accept the 4 params `prefix`, `startDate`, `endDate` and `limit`and we get proper response.
Screenshot attached.![Swagger_Error](/uploads/0f1b760a28aa3501b4a88acef12cfbfd/Swagger_Error.JPG)