Ingestion Workflow issueshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues2021-05-11T15:34:21Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/117Ingestion process - Differentiate status "Success" to reflect true success an...2021-05-11T15:34:21ZDebasis ChatterjeeIngestion process - Differentiate status "Success" to reflect true success and partial successRight now, DAG status may show "Success" and colored green in graph/tree view although inside the log we can see some failures. Such as when 10 IDs created and 2 IDs rejected.
Propose two changes - Status - "Completed with some failures...Right now, DAG status may show "Success" and colored green in graph/tree view although inside the log we can see some failures. Such as when 10 IDs created and 2 IDs rejected.
Propose two changes - Status - "Completed with some failures" instead of "Success" if some or all records could not be processed for some reason.
In Airflow graph/tree view - perhaps use different shade of green.
Copy of old ticket
https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/-/issues/171#note_37091
cc - @Keith_Wall for information.https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/116Airflow Experimental Delete API not implemented for RBAC enabled Airflow.2021-06-14T16:39:57ZMayank Saggar [Microsoft]Airflow Experimental Delete API not implemented for RBAC enabled Airflow.There is a issue with Airflow where if RBAC for Webserver is enabled, the delete API from airflow was returning 404. On digging the airflow source code, it was discovered that there is separate handler for api requests with rbac enabled ...There is a issue with Airflow where if RBAC for Webserver is enabled, the delete API from airflow was returning 404. On digging the airflow source code, it was discovered that there is separate handler for api requests with rbac enabled which doesn't provide delete API.
Github Source: [endpoints when rbac enabled](https://github.com/apache/airflow/blob/1.10.12/airflow/www_rbac/api/experimental/endpoints.py)
[endpoints when rbac disabled](https://github.com/apache/airflow/blob/1.10.12/airflow/www/api/experimental/endpoints.py)
So if we remove RBAC from webserver we get the Experimental delete API.https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/115POST v1/workflow (Create Workflow) with no workflow name in request body thro...2022-07-27T15:51:11ZAalekh JainPOST v1/workflow (Create Workflow) with no workflow name in request body throws 500 internal server error## Description
**Current Behaviour**
For the given request body
```json
{
"description": "This prints a storage record sent to the system",
"registrationInstructions": {
"dagName": "hello"
}
}
```
Error thrown is
```json
...## Description
**Current Behaviour**
For the given request body
```json
{
"description": "This prints a storage record sent to the system",
"registrationInstructions": {
"dagName": "hello"
}
}
```
Error thrown is
```json
{
"code": 500,
"reason": "Unexpectedly failed to insert item into CosmosDB",
"message": "[\"The input name 'null' is invalid. Ensure to provide a unique non-empty string less than '1024' characters.\"], {\"userAgent\":\"azsdk-java-cosmos/4.7.1 Windows10/10.0 JRE/1.8.0_265\",\"requestLatencyInMs\":201,\"requestStartTimeUTC\":\"2021-04-21T11:02:33.038Z\",\"requestEndTimeUTC\":\"2021-04-21T11:02:33.239Z\",\"connectionMode\":\"DIRECT\",\"responseStatisticsList\":[{\"storeResult\":{\"storePhysicalAddress\":\"rntbd://cdb-ms-prod-eastus2-fd7.documents.azure.com:14306/apps/a78846d5-27aa-45e8-bef0-0950c8a3c1d2/services/9a98fb60-a0fb-43e4-be05-3fe8dc8d6498/partitions/e158dec0-4caf-42e6-b7e8-1eb9dc9b7c84/replicas/132618830238230982p/\",\"lsn\":84828,\"globalCommittedLsn\":84828,\"partitionKeyRangeId\":\"1\",\"isValid\":true,\"statusCode\":400,\"subStatusCode\":0,\"isGone\":false,\"isNotFound\":false,\"isInvalidPartition\":false,\"requestCharge\":1.24,\"itemLSN\":-1,\"sessionToken\":\"-1#84828\",\"exception\":\"[\\\"The input name 'null' is invalid. Ensure to provide a unique non-empty string less than '1024' characters.\\\"]\",\"transportRequestTimeline\":[{\"eventName\":\"created\",\"durationInMicroSec\":\"0\",\"startTime\":\"2021-04-21T11:02:33.039Z\"},{\"eventName\":\"queued\",\"durationInMicroSec\":\"0\",\"startTime\":\"2021-04-21T11:02:33.039Z\"},{\"eventName\":\"channelAcquisitionStarted\",\"durationInMicroSec\":\"0\",\"startTime\":\"2021-04-21T11:02:33.039Z\"},{\"eventName\":\"pipelined\",\"durationInMicroSec\":\"0\",\"startTime\":\"2021-04-21T11:02:33.039Z\"},{\"eventName\":\"transitTime\",\"durationInMicroSec\":\"200000\",\"startTime\":\"2021-04-21T11:02:33.039Z\"},{\"eventName\":\"received\",\"durationInMicroSec\":\"0\",\"startTime\":\"2021-04-21T11:02:33.239Z\"},{\"eventName\":\"completed\",\"durationInMicroSec\":\"0\",\"startTime\":\"2021-04-21T11:02:33.239Z\"}],\"rntbdRequestLengthInBytes\":713,\"rntbdResponseLengthInBytes\":329,\"requestPayloadLengthInBytes\":296,\"responsePayloadLengthInBytes\":null,\"channelTaskQueueSize\":1,\"pendingRequestsCount\":1,\"serviceEndpointStatistics\":{\"availableChannels\":1,\"acquiredChannels\":0,\"executorTaskQueueSize\":0,\"inflightRequests\":1,\"lastSuccessfulRequestTime\":\"2021-04-21T11:00:24.327Z\",\"lastRequestTime\":\"2021-04-21T11:00:24.121Z\",\"createdTime\":\"2021-04-21T10:54:28.705Z\",\"isClosed\":false}},\"requestResponseTimeUTC\":\"2021-04-21T11:02:33.239Z\",\"requestResourceType\":\"Document\",\"requestOperationType\":\"Create\"}],\"supplementalResponseStatisticsList\":[],\"addressResolutionStatistics\":{},\"regionsContacted\":[\"https://osdu-mvp-dp1dev-qs29-db-eastus2.documents.azure.com:443/\"],\"retryContext\":{\"retryCount\":0,\"statusAndSubStatusCodes\":null,\"retryLatency\":0},\"metadataDiagnosticsContext\":{\"metadataDiagnosticList\":null},\"serializationDiagnosticsContext\":{\"serializationDiagnosticsList\":[{\"serializationType\":\"ITEM_SERIALIZATION\",\"startTimeUTC\":\"2021-04-21T11:02:33.038Z\",\"endTimeUTC\":\"2021-04-21T11:02:33.039Z\",\"durationInMicroSec\":1000}]},\"gatewayStatistics\":null,\"systemInformation\":{\"usedMemory\":\"159620 KB\",\"availableMemory\":\"3510396 KB\",\"systemCpuLoad\":\"(2021-04-21T11:02:03.920Z 5.4%), (2021-04-21T11:02:08.920Z 5.1%), (2021-04-21T11:02:13.922Z 7.7%), (2021-04-21T11:02:18.922Z 14.8%), (2021-04-21T11:02:23.920Z 8.5%), (2021-04-21T11:02:28.921Z 7.8%)\"},\"clientCfgs\":{\"id\":0,\"numberOfClients\":1,\"connCfg\":{\"rntbd\":\"(cto:PT5S, rto:PT5S, icto:PT0S, ieto:PT1H, mcpe:130, mrpc:30)\",\"gw\":\"(cps:1000, rto:PT5S, icto:null, p:false)\",\"other\":\"(ed: true, cs: false)\"},\"consistencyCfg\":\"(consistency: null, mm: true, prgns: [])\"}}"
}
```
**Expected Behaviour**
Should throw an error saying `workflowName` not present in request body.
cc: @kibattul @vineethgunahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/114POST /v1/workflow/{workflow_name}/workflowRun (Trigger Workflow) with empty r...2021-06-14T16:18:30ZAalekh JainPOST /v1/workflow/{workflow_name}/workflowRun (Trigger Workflow) with empty runId throws 500 internal server error## Description
**Current Behaviour**
For the given request body
```json
{
"runId": "",
"executionContext": {
}
}
```
Error thrown is
```json
{
"code": 500,
"reason": "Unexpectedly failed to insert item into CosmosDB",
...## Description
**Current Behaviour**
For the given request body
```json
{
"runId": "",
"executionContext": {
}
}
```
Error thrown is
```json
{
"code": 500,
"reason": "Unexpectedly failed to insert item into CosmosDB",
"message": "[\"The input name '' is invalid. Ensure to provide a unique non-empty string less than '1024' characters.\"], {\"userAgent\":\"azsdk-java-cosmos/4.7.1 Windows10/10.0 JRE/1.8.0_265\",\"requestLatencyInMs\":212,\"requestStartTimeUTC\":\"2021-04-19T09:01:33.929Z\",\"requestEndTimeUTC\":\"2021-04-19T09:01:34.141Z\",\"connectionMode\":\"DIRECT\",\"responseStatisticsList\":[{\"storeResult\":{\"storePhysicalAddress\":\"rntbd://cdb-ms-prod-eastus2-fd7.documents.azure.com:14178/apps/a78846d5-27aa-45e8-bef0-0950c8a3c1d2/services/9a98fb60-a0fb-43e4-be05-3fe8dc8d6498/partitions/e158dec0-4caf-42e6-b7e8-1eb9dc9b7c84/replicas/132593652864958972p/\",\"lsn\":84405,\"globalCommittedLsn\":84405,\"partitionKeyRangeId\":\"1\",\"isValid\":true,\"statusCode\":400,\"subStatusCode\":0,\"isGone\":false,\"isNotFound\":false,\"isInvalidPartition\":false,\"requestCharge\":1.24,\"itemLSN\":-1,\"sessionToken\":\"-1#84405\",\"exception\":\"[\\\"The input name '' is invalid. Ensure to provide a unique non-empty string less than '1024' characters.\\\"]\",\"transportRequestTimeline\":[{\"eventName\":\"created\",\"durationInMicroSec\":\"0\",\"startTime\":\"2021-04-19T09:01:33.931Z\"},{\"eventName\":\"queued\",\"durationInMicroSec\":\"0\",\"startTime\":\"2021-04-19T09:01:33.931Z\"},{\"eventName\":\"channelAcquisitionStarted\",\"durationInMicroSec\":\"3000\",\"startTime\":\"2021-04-19T09:01:33.931Z\"},{\"eventName\":\"pipelined\",\"durationInMicroSec\":\"1000\",\"startTime\":\"2021-04-19T09:01:33.934Z\"},{\"eventName\":\"transitTime\",\"durationInMicroSec\":\"204000\",\"startTime\":\"2021-04-19T09:01:33.935Z\"},{\"eventName\":\"received\",\"durationInMicroSec\":\"1000\",\"startTime\":\"2021-04-19T09:01:34.139Z\"},{\"eventName\":\"completed\",\"durationInMicroSec\":\"1000\",\"startTime\":\"2021-04-19T09:01:34.140Z\"}],\"rntbdRequestLengthInBytes\":714,\"rntbdResponseLengthInBytes\":325,\"requestPayloadLengthInBytes\":282,\"responsePayloadLengthInBytes\":null,\"channelTaskQueueSize\":1,\"pendingRequestsCount\":1,\"serviceEndpointStatistics\":{\"availableChannels\":1,\"acquiredChannels\":0,\"executorTaskQueueSize\":0,\"inflightRequests\":1,\"lastSuccessfulRequestTime\":\"2021-04-19T08:52:54.424Z\",\"lastRequestTime\":\"2021-04-19T08:52:54.211Z\",\"createdTime\":\"2021-04-19T08:34:55.741Z\",\"isClosed\":false}},\"requestResponseTimeUTC\":\"2021-04-19T09:01:34.141Z\",\"requestResourceType\":\"Document\",\"requestOperationType\":\"Create\"}],\"supplementalResponseStatisticsList\":[],\"addressResolutionStatistics\":{},\"regionsContacted\":[\"https://osdu-mvp-dp1dev-qs29-db-eastus2.documents.azure.com:443/\"],\"retryContext\":{\"retryCount\":0,\"statusAndSubStatusCodes\":null,\"retryLatency\":0},\"metadataDiagnosticsContext\":{\"metadataDiagnosticList\":null},\"serializationDiagnosticsContext\":{\"serializationDiagnosticsList\":[{\"serializationType\":\"ITEM_SERIALIZATION\",\"startTimeUTC\":\"2021-04-19T09:01:33.929Z\",\"endTimeUTC\":\"2021-04-19T09:01:33.929Z\",\"durationInMicroSec\":0}]},\"gatewayStatistics\":null,\"systemInformation\":{\"usedMemory\":\"237293 KB\",\"availableMemory\":\"3432723 KB\",\"systemCpuLoad\":\"(2021-04-19T09:01:08.919Z 5.1%), (2021-04-19T09:01:13.920Z 6.9%), (2021-04-19T09:01:18.921Z 4.0%), (2021-04-19T09:01:23.919Z 5.1%), (2021-04-19T09:01:28.920Z 5.3%), (2021-04-19T09:01:33.922Z 10.5%)\"},\"clientCfgs\":{\"id\":0,\"numberOfClients\":1,\"connCfg\":{\"rntbd\":\"(cto:PT5S, rto:PT5S, icto:PT0S, ieto:PT1H, mcpe:130, mrpc:30)\",\"gw\":\"(cps:1000, rto:PT5S, icto:null, p:false)\",\"other\":\"(ed: true, cs: false)\"},\"consistencyCfg\":\"(consistency: null, mm: true, prgns: [])\"}}"
}
```
**Expected Behaviour**
Should throw an error saying `runId` cannot be empty or invalid `runId` given OR `runId` needs to be generated (similar to what happens when `runId` field is not present in the request body) - Need confirmation on the expected behaviour
Works fine when the request body does not have `runId` as the key
```json
{
"executionContext": {
}
}
```
cc: @kibattul @vineethgunahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/113[core] Validating mandatory headers in authorization filter2021-04-09T09:27:50ZAalekh Jain[core] Validating mandatory headers in authorization filterThe validation for mandatory headers, such as `data-partition-id` and `authorization` is **not** present in `AuthorizationFilter`. This check needs to be added to core.
For reference: [Schema service validation of mandatory headers](ht...The validation for mandatory headers, such as `data-partition-id` and `authorization` is **not** present in `AuthorizationFilter`. This check needs to be added to core.
For reference: [Schema service validation of mandatory headers](https://community.opengroup.org/osdu/platform/system/schema-service/-/blob/master/schema-core/src/main/java/org/opengroup/osdu/schema/security/AuthorizationFilter.java#L55)
Original issue: #96
cc: @kibattul @msrivastavahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/112Swagger is not behaving correctly for the API /v1/workflow/{workflow_name}/wo...2021-04-07T12:08:26ZMonalisa SrivastavaSwagger is not behaving correctly for the API /v1/workflow/{workflow_name}/workflowRun getAllRunInstancesSwagger https://osdu-glab.msft-osdu-test.org/api/workflow/swagger-ui.html#/workflow-run-api/getAllRunInstancesUsingGET is not accepting params however through postman the API works fine and accept the 4 params `prefix`, `startDate`, `end...Swagger https://osdu-glab.msft-osdu-test.org/api/workflow/swagger-ui.html#/workflow-run-api/getAllRunInstancesUsingGET is not accepting params however through postman the API works fine and accept the 4 params `prefix`, `startDate`, `endDate` and `limit`and we get proper response.
Screenshot attached.![Swagger_Error](/uploads/0f1b760a28aa3501b4a88acef12cfbfd/Swagger_Error.JPG)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/111/v1/workflow List all the workflow applicable for a tenant is not working f...2021-04-06T17:21:33ZKamlesh Todai/v1/workflow List all the workflow applicable for a tenant is not working for AWS, Azure and IBMThe endpoint to list all the workflows applicable for a tenant is not working on AWS, Azure, and IBM. It is working on GCP
Azure
GET https://osdu-demo.msft-osdu-test.org/v1/workflow/ Status 404 Not Found
IBM
GET https://osdu-cpd-osdu.odi...The endpoint to list all the workflows applicable for a tenant is not working on AWS, Azure, and IBM. It is working on GCP
Azure
GET https://osdu-demo.msft-osdu-test.org/v1/workflow/ Status 404 Not Found
IBM
GET https://osdu-cpd-osdu.odi-osdu-og-fa7661852f2ab29a6be32f560b2f5573-0000.us-south.containers.appdomain.cloud/osdu-workflow/v1/workflow/ Status 404 Not Found
AWS
GET https://api.forumtesting.osdu.aws/v1/workflow/ Status 404 Not Found
GCP
GET https://workflow-drgfbg5txq-uc.a.run.app/v1/workflow/ Status 200 OKhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/110No error message GET /v1/workflow/{workflow_name} getWorkflowByName for broke...2022-06-30T20:24:26ZMonalisa SrivastavaNo error message GET /v1/workflow/{workflow_name} getWorkflowByName for broken or incorrect DAGsTested on Azure:
Steps followed :
1) Register a DAG\Workflow using the POST /workflow create API
2) Use the GET /workflow/{id} getWorkflowById to check the status.
Expected Result : If workflow is created successfully the information ...Tested on Azure:
Steps followed :
1) Register a DAG\Workflow using the POST /workflow create API
2) Use the GET /workflow/{id} getWorkflowById to check the status.
Expected Result : If workflow is created successfully the information should be displayed for the Workflow
Actual Result : Even if the Workflow is broken and not created successfully, the API displays no error, AIrflow shows broken DAG error.https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/109workflow create API returns 500 instead of 400 for multiple cases in the requ...2021-03-25T11:09:48ZMonalisa Srivastavaworkflow create API returns 500 instead of 400 for multiple cases in the request bodyTested on Azure : API - https://{HOST}/api/workflow/v1/workflow
For the workflow create API with Empty/missing workflowName in the payload
We are getting the following error :
{{"500 Error: Internal Server Error
{
""code"": 500,
"...Tested on Azure : API - https://{HOST}/api/workflow/v1/workflow
For the workflow create API with Empty/missing workflowName in the payload
We are getting the following error :
{{"500 Error: Internal Server Error
{
""code"": 500,
""reason"": ""Unexpectedly failed to insert item into CosmosDB"",
""message"": ""[\""The input name 'null' is invalid. Ensure to provide a unique non-empty string less than '1024' characters.\""], {\""userAgent\"":\""azsdk-java-cosmos/4.7.1 Linux/5.4.0-1035-azure JRE/1.8.0_212\"",\""requestLatencyInMs\"":169,\""requestStartTimeUTC\"":\""2021-03-01T10:48:11.806Z\"",\""requestEndTimeUTC\"":\""2021-03-01T10:48:11.975Z\"",\""connectionMode\"":\""DIRECT\"",\""responseStatisticsList\"":[{\""storeResult\"":{\""storePhysicalAddress\"":\""rntbd://cdb-ms-prod-eastus2-fd29.documents.azure.com:14331/apps/b0b676c7-2ade-4d27-aca1-4fc7bbdeb696/services/19ba9dad-08cb-4507-99e3-86fd380f68ef/partitions/c2edaa6d-10b5-4b3e-b211-6ecef229dafe/replicas/132557659531536721p/\"",\""lsn\"":5263519,\""globalCommittedLsn\"":5263519,\""partitionKeyRangeId\"":\""1\"",\""isValid\"":true,\""statusCode\"":400,\""subStatusCode\"":0,\""isGone\"":false,\""isNotFound\"":false,\""isInvalidPartition\"":false,\""requestCharge\"":1.24,\""itemLSN\"":-1,\""sessionToken\"":\""-1#5263519\"",\""exception\"":\""[\\\""The input name 'null' is invalid. Ensure to provide a unique non-empty string less than '1024' characters.\\\""]\"",\""transportRequestTimeline\"":[{\""eventName\"":\""created\"",\""durationInMicroSec\"":\""1000\"",\""startTime\"":\""2021-03-01T10:48:11.806Z\""},{\""eventName\"":\""queued\"",\""durationInMicroSec\"":\""0\"",\""startTime\"":\""2021-03-01T10:48:11.807Z\""},{\""eventName\"":\""channelAcquisitionStarted\"",\""durationInMicroSec\"":\""137000\"",\""startTime\"":\""2021-03-01T10:48:11.807Z\""},{\""eventName\"":\""pipelined\"",\""durationInMicroSec\"":\""1000\"",\""startTime\"":\""2021-03-01T10:48:11.944Z\""},{\""eventName\"":\""transitTime\"",\""durationInMicroSec\"":\""29000\"",\""startTime\"":\""2021-03-01T10:48:11.945Z\""},{\""eventName\"":\""received\"",\""durationInMicroSec\"":\""1000\"",\""startTime\"":\""2021-03-01T10:48:11.974Z\""},{\""eventName\"":\""completed\"",\""durationInMicroSec\"":\""1000\"",\""startTime\"":\""2021-03-01T10:48:11.975Z\""}],\""rntbdRequestLengthInBytes\"":706,\""rntbdResponseLengthInBytes\"":331,\""requestPayloadLengthInBytes\"":289,\""responsePayloadLengthInBytes\"":null,\""channelTaskQueueSize\"":1,\""pendingRequestsCount\"":1,\""serviceEndpointStatistics\"":{\""availableChannels\"":0,\""acquiredChannels\"":0,\""executorTaskQueueSize\"":0,\""inflightRequests\"":1,\""lastSuccessfulRequestTime\"":\""2021-03-01T10:48:11.806Z\"",\""lastRequestTime\"":\""2021-03-01T10:48:11.806Z\"",\""createdTime\"":\""2021-03-01T10:48:11.807Z\"",\""isClosed\"":false}},\""requestResponseTimeUTC\"":\""2021-03-01T10:48:11.975Z\"",\""requestResourceType\"":\""Document\"",\""requestOperationType\"":\""Create\""}],\""supplementalResponseStatisticsList\"":[],\""addressResolutionStatistics\"":{},\""regionsContacted\"":[\""https://osdu-r3mvp-dp1dev-077y-db-eastus2.documents.azure.com:443/\""],\""retryContext\"":{\""retryCount\"":0,\""statusAndSubStatusCodes\"":null,\""retryLatency\"":0},\""metadataDiagnosticsContext\"":{\""metadataDiagnosticList\"":null},\""serializationDiagnosticsContext\"":{\""serializationDiagnosticsList\"":[{\""serializationType\"":\""ITEM_SERIALIZATION\"",\""startTimeUTC\"":\""2021-03-01T10:48:11.806Z\"",\""endTimeUTC\"":\""2021-03-01T10:48:11.806Z\"",\""durationInMicroSec\"":0}]},\""gatewayStatistics\"":null,\""systemInformation\"":{\""usedMemory\"":\""179848 KB\"",\""availableMemory\"":\""7768824 KB\"",\""systemCpuLoad\"":\""(2021-03-01T10:47:44.536Z 7.3%), (2021-03-01T10:47:49.536Z 7.1%), (2021-03-01T10:47:54.536Z 6.9%), (2021-03-01T10:47:59.536Z 8.8%), (2021-03-01T10:48:04.536Z 7.1%), (2021-03-01T10:48:09.536Z 7.3%)\""},\""clientCfgs\"":{\""id\"":0,\""numberOfClients\"":1,\""connCfg\"":{\""rntbd\"":\""(cto:PT5S, rto:PT5S, icto:PT0S, ieto:PT1H, mcpe:130, mrpc:30)\"",\""gw\"":\""(cps:1000, rto:PT5S, icto:null, p:false)\"",\""other\"":\""(ed: true, cs: false)\""},\""consistencyCfg\"":\""(consistency: null, mm: true, prgns: [])\""}}""
}"}}https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/108registerCustomOperator API </customOperator> returns success if an empty "pro...2022-06-30T20:24:49ZMonalisa SrivastavaregisterCustomOperator API </customOperator> returns success if an empty "properties" block is passed but throws 400 if "properties" block is missingTested on AZURE : API : https://{HOST}/api/workflow/v1/customOperator
Hit the API with below payload:
{
"name": "string",
"className": "string",
"description": "string",
"content": "string",
"properties": [
]
}
Respons...Tested on AZURE : API : https://{HOST}/api/workflow/v1/customOperator
Hit the API with below payload:
{
"name": "string",
"className": "string",
"description": "string",
"content": "string",
"properties": [
]
}
Response : API returns success 200 OK
Now hit the same API with Properties block missing and observe that 400 response code is returned
Expected
If properties block is mandatory, then empty block should also return 400 instead of 200 and return a message with the exact reasonhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/107POST https://{HOST}/api/workflow/v1/customOperator API should return error me...2021-03-25T10:11:46ZMonalisa SrivastavaPOST https://{HOST}/api/workflow/v1/customOperator API should return error message with proper reason for 400 Bad Request errorsTested on Azure : API POST https://{HOST}/api/workflow/v1/customOperator
Below are the scenarios for which the API returns 400 (Bad Request ) Response
Name key is missing or empty
ClassName key is missing or empty
content key is missin...Tested on Azure : API POST https://{HOST}/api/workflow/v1/customOperator
Below are the scenarios for which the API returns 400 (Bad Request ) Response
Name key is missing or empty
ClassName key is missing or empty
content key is missing or empty
properties block is missing
Expected
THe API should also return a valid message mentioning the exact reason for BadRequesthttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/106[Draft] [WIP] Integration Tests [core]2021-03-23T14:14:08ZAalekh Jain[Draft] [WIP] Integration Tests [core]Integration tests in core do not have sufficient coverage. This issue aims at adding sufficient integrations tests in core.Integration tests in core do not have sufficient coverage. This issue aims at adding sufficient integrations tests in core.https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/105GET /v1/workflow/{workflow_name}/workflowRun getAllRunInstances parameters2023-11-14T14:35:21ZAalekh JainGET /v1/workflow/{workflow_name}/workflowRun getAllRunInstances parameters1. Please specify the expected behaviour from the following parameters present in the api spec for get all run instances of a workflow
* [boolean] **partial**
* [string] **conf**
2. What is the expected format in which the start dat...1. Please specify the expected behaviour from the following parameters present in the api spec for get all run instances of a workflow
* [boolean] **partial**
* [string] **conf**
2. What is the expected format in which the start date and end date is provided. Is it going to be Unix epoch time? In case it's going to be unix epoch time, maybe it's better to name it `startTimeStamp` rather than `startDate` and `endTimeStamp` rather than `endDate`. A minor nit, the type of `endDate` is mentioned as boolean in the api spec. Should be string.
3. Currently, the response only returns the list of workflow runs and the cursor for the subsequent requests is not being returned. In such a scenario, how can one obtain the cursor for the subsequent request. Shouldn't we pass the cursor as part of the response? How do we plan to handle this scenario?
4. What is the expected behaviour when an invalid parameter is passed (apart from `limit`, `cursor`, `startDate`, `endDate` etc.)?
cc: @kibattulhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/103Put workflowRun as value for Airflow DAG run_id2021-03-17T09:21:26ZSiarhei Khaletski (EPAM)Put workflowRun as value for Airflow DAG run_idhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/102POST /v1/workflow with blank\without the workflowName given 500 internal serv...2021-06-10T11:25:09ZMonalisa SrivastavaPOST /v1/workflow with blank\without the workflowName given 500 internal server errorPOST /v1/workflow with blank\null workflowName given 500 internal server error
We should get proper error empty workflow_name or workflow name is required.
If we do not provide the workflow_name key in the payload we get the following ...POST /v1/workflow with blank\null workflowName given 500 internal server error
We should get proper error empty workflow_name or workflow name is required.
If we do not provide the workflow_name key in the payload we get the following error:
{{"500 Error: Internal Server Error
{
""code"": 500,
""reason"": ""Unexpectedly failed to insert item into CosmosDB"",
""message"": ""[\""The input name 'null' is invalid. Ensure to provide a unique non-empty string less than '1024' characters.\""], {\""userAgent\"":\""azsdk-java-cosmos/4.7.1 Linux/5.4.0-1035-azure JRE/1.8.0_212\"",\""requestLatencyInMs\"":169,\""requestStartTimeUTC\"":\""2021-03-01T10:48:11.806Z\"",\""requestEndTimeUTC\"":\""2021-03-01T10:48:11.975Z\"",\""connectionMode\"":\""DIRECT\"",\""responseStatisticsList\"":[{\""storeResult\"":{\""storePhysicalAddress\"":\""rntbd://cdb-ms-prod-eastus2-fd29.documents.azure.com:14331/apps/b0b676c7-2ade-4d27-aca1-4fc7bbdeb696/services/19ba9dad-08cb-4507-99e3-86fd380f68ef/partitions/c2edaa6d-10b5-4b3e-b211-6ecef229dafe/replicas/132557659531536721p/\"",\""lsn\"":5263519,\""globalCommittedLsn\"":5263519,\""partitionKeyRangeId\"":\""1\"",\""isValid\"":true,\""statusCode\"":400,\""subStatusCode\"":0,\""isGone\"":false,\""isNotFound\"":false,\""isInvalidPartition\"":false,\""requestCharge\"":1.24,\""itemLSN\"":-1,\""sessionToken\"":\""-1#5263519\"",\""exception\"":\""[\\\""The input name 'null' is invalid. Ensure to provide a unique non-empty string less than '1024' characters.\\\""]\"",\""transportRequestTimeline\"":[{\""eventName\"":\""created\"",\""durationInMicroSec\"":\""1000\"",\""startTime\"":\""2021-03-01T10:48:11.806Z\""},{\""eventName\"":\""queued\"",\""durationInMicroSec\"":\""0\"",\""startTime\"":\""2021-03-01T10:48:11.807Z\""},{\""eventName\"":\""channelAcquisitionStarted\"",\""durationInMicroSec\"":\""137000\"",\""startTime\"":\""2021-03-01T10:48:11.807Z\""},{\""eventName\"":\""pipelined\"",\""durationInMicroSec\"":\""1000\"",\""startTime\"":\""2021-03-01T10:48:11.944Z\""},{\""eventName\"":\""transitTime\"",\""durationInMicroSec\"":\""29000\"",\""startTime\"":\""2021-03-01T10:48:11.945Z\""},{\""eventName\"":\""received\"",\""durationInMicroSec\"":\""1000\"",\""startTime\"":\""2021-03-01T10:48:11.974Z\""},{\""eventName\"":\""completed\"",\""durationInMicroSec\"":\""1000\"",\""startTime\"":\""2021-03-01T10:48:11.975Z\""}],\""rntbdRequestLengthInBytes\"":706,\""rntbdResponseLengthInBytes\"":331,\""requestPayloadLengthInBytes\"":289,\""responsePayloadLengthInBytes\"":null,\""channelTaskQueueSize\"":1,\""pendingRequestsCount\"":1,\""serviceEndpointStatistics\"":{\""availableChannels\"":0,\""acquiredChannels\"":0,\""executorTaskQueueSize\"":0,\""inflightRequests\"":1,\""lastSuccessfulRequestTime\"":\""2021-03-01T10:48:11.806Z\"",\""lastRequestTime\"":\""2021-03-01T10:48:11.806Z\"",\""createdTime\"":\""2021-03-01T10:48:11.807Z\"",\""isClosed\"":false}},\""requestResponseTimeUTC\"":\""2021-03-01T10:48:11.975Z\"",\""requestResourceType\"":\""Document\"",\""requestOperationType\"":\""Create\""}],\""supplementalResponseStatisticsList\"":[],\""addressResolutionStatistics\"":{},\""regionsContacted\"":[\""https://osdu-r3mvp-dp1dev-077y-db-eastus2.documents.azure.com:443/\""],\""retryContext\"":{\""retryCount\"":0,\""statusAndSubStatusCodes\"":null,\""retryLatency\"":0},\""metadataDiagnosticsContext\"":{\""metadataDiagnosticList\"":null},\""serializationDiagnosticsContext\"":{\""serializationDiagnosticsList\"":[{\""serializationType\"":\""ITEM_SERIALIZATION\"",\""startTimeUTC\"":\""2021-03-01T10:48:11.806Z\"",\""endTimeUTC\"":\""2021-03-01T10:48:11.806Z\"",\""durationInMicroSec\"":0}]},\""gatewayStatistics\"":null,\""systemInformation\"":{\""usedMemory\"":\""179848 KB\"",\""availableMemory\"":\""7768824 KB\"",\""systemCpuLoad\"":\""(2021-03-01T10:47:44.536Z 7.3%), (2021-03-01T10:47:49.536Z 7.1%), (2021-03-01T10:47:54.536Z 6.9%), (2021-03-01T10:47:59.536Z 8.8%), (2021-03-01T10:48:04.536Z 7.1%), (2021-03-01T10:48:09.536Z 7.3%)\""},\""clientCfgs\"":{\""id\"":0,\""numberOfClients\"":1,\""connCfg\"":{\""rntbd\"":\""(cto:PT5S, rto:PT5S, icto:PT0S, ieto:PT1H, mcpe:130, mrpc:30)\"",\""gw\"":\""(cps:1000, rto:PT5S, icto:null, p:false)\"",\""other\"":\""(ed: true, cs: false)\""},\""consistencyCfg\"":\""(consistency: null, mm: true, prgns: [])\""}}""
}"}}Aalekh JainAalekh Jainhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/101GET /v1/workflow getAllWorkflowsForTenant fails with 500 Internal server error2021-04-08T10:22:17ZMonalisa SrivastavaGET /v1/workflow getAllWorkflowsForTenant fails with 500 Internal server errorGET /v1/workflow getAllWorkflowsForTenant fails with 500 Internal server error
Actual result :
{
"timestamp": 1614585621023,
"status": 500,
"error": "Internal Server Error",
"message": "getAllWorkflowForTenant is not implemented...GET /v1/workflow getAllWorkflowsForTenant fails with 500 Internal server error
Actual result :
{
"timestamp": 1614585621023,
"status": 500,
"error": "Internal Server Error",
"message": "getAllWorkflowForTenant is not implemented for Azure",
"path": "/api/workflow/v1/workflow"
}
Expected result : we should get the list of all the workflows registered for the data partition.Monalisa SrivastavaMonalisa Srivastavahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/100PUT v1/workflow/{workflow_name}/workflowRun/{runId} doesnt update the status ...2021-05-25T04:51:57ZMonalisa SrivastavaPUT v1/workflow/{workflow_name}/workflowRun/{runId} doesnt update the status of the workflow on the Airflow UIPUT v1/workflow/{workflow_name}/workflowRun/{runId} doesnt update the status of the workflow,
When we try to update the status for a workflow ,though we get a 200 Ok and success response the Airflow still shows the same status.
Howeve...PUT v1/workflow/{workflow_name}/workflowRun/{runId} doesnt update the status of the workflow,
When we try to update the status for a workflow ,though we get a 200 Ok and success response the Airflow still shows the same status.
However from Airflow UI, when we update the status, the status changes suucessfully.Monalisa SrivastavaMonalisa Srivastavahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/99correlation-id is missing in the workflow service API's response2021-06-10T11:25:29ZMonalisa Srivastavacorrelation-id is missing in the workflow service API's responsecorrelation-id is missing in the workflow execution response like sample response from schema service :
Sample response from workflow service :
```
cache-control: no-cache, no-store, max-age=0, must-revalidate
connection: keep-alive
...correlation-id is missing in the workflow execution response like sample response from schema service :
Sample response from workflow service :
```
cache-control: no-cache, no-store, max-age=0, must-revalidate
connection: keep-alive
content-security-policy: default-src 'self'
content-type: application/json
date: Wed, 03 Mar 2021 11:52:13 GMT
expires: 0
pragma: no-cache
request-context: appId=cid-v1:9f744e47-15f4-45a1-9703-d398e09cbbb5
server: istio-envoy
strict-transport-security: max-age=31536000 ; includeSubDomains
transfer-encoding: chunked
x-content-type-options: nosniff
x-envoy-decorator-operation: workflow.osdu.svc.cluster.local:80/*
x-envoy-upstream-service-time: 152
x-frame-options: DENY
x-xss-protection: 1; mode=block
```
Sample response from schema service :
```
{
"access-control-allow-credentials": "[true]",
"access-control-allow-headers": "[origin, content-type, accept, authorization, data-partition-id, correlation-id, appkey]",
"access-control-allow-methods": "[GET, POST, PUT, DELETE, OPTIONS, HEAD, PATCH]",
"access-control-allow-origin": "[*]",
"cache-control": "[no-cache, no-store, must-revalidate]",
"connection": "keep-alive",
"content-security-policy": "[default-src 'self']",
"content-type": "application/json",
"correlation-id": "fbda9c60-fbbf-43fe-bdc4-550e1a17e5e6",
"date": "Wed, 03 Mar 2021 11:59:45 GMT",
"expires": "[0]",
"request-context": "appId=cid-v1:9f744e47-15f4-45a1-9703-d398e09cbbb5",
"server": "istio-envoy",
"strict-transport-security": "[max-age=31536000; includeSubDomains]",
"transfer-encoding": "chunked",
"x-content-type-options": "[nosniff]",
"x-envoy-decorator-operation": "schema-service.osdu.svc.cluster.local:80/*",
"x-envoy-upstream-service-time": "6312",
"x-frame-options": "DENY",
"x-xss-protection": "[1; mode=block]"
}
```Aalekh JainAalekh Jainhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/98GET /v1/workflow/{workflow_name}/workflowRun getAllRunInstances doesnt respon...2021-04-09T13:07:56ZMonalisa SrivastavaGET /v1/workflow/{workflow_name}/workflowRun getAllRunInstances doesnt respond correctlyGET /v1/workflow/{workflow_name}/workflowRun getAllRunInstances reqs params even with blank json it gives a 200 OK however no details in response
Following filters are also missing :
String prefix = (String) params.get("prefix");
String ...GET /v1/workflow/{workflow_name}/workflowRun getAllRunInstances reqs params even with blank json it gives a 200 OK however no details in response
Following filters are also missing :
String prefix = (String) params.get("prefix");
String startDate = (String) params.get("startDate");
String endDate = (String) params.get("endDate");
String limit = (String) params.get("limit");Kishore BattulaAalekh JainKishore Battulahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/97DELETE /v1/workflow/{workflow_name}deleteWorkflowById fails to delete workflo...2022-08-23T11:19:22ZMonalisa SrivastavaDELETE /v1/workflow/{workflow_name}deleteWorkflowById fails to delete workflow which have been executedDELETE /v1/workflow/{workflow_name}deleteWorkflowById fails to delete workflow which have been executed, however the newly created workflow which have not been executed gets deleted successfully.
Actual Result : we get the following err...DELETE /v1/workflow/{workflow_name}deleteWorkflowById fails to delete workflow which have been executed, however the newly created workflow which have not been executed gets deleted successfully.
Actual Result : we get the following error :
{
"timestamp": 1615385317197,
"status": 404,
"error": "Not Found",
"message": "Workflow: csv_OneStep_wf doesn't exist",
"path": "/api/workflow/v1/workflow/csv_OneStep_wf"
}
Expected Result : The workflow should be delete successfully
Also please note though in the API we are mentioning to use the {workflow_name} description says deleteWorkflowById which should be corrected.Aalekh JainMonalisa SrivastavaAalekh Jain