Data Ingestion issueshttps://community.opengroup.org/groups/osdu/platform/data-flow/ingestion/-/issues2023-10-04T02:22:40Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser-v2/-/issues/13M20/Azure/Preship - failure handling Trajectory data2023-10-04T02:22:40ZDebasis ChatterjeeM20/Azure/Preship - failure handling Trajectory dataUsed this source data[Trajectory-WITSML-DC.txt](/uploads/9732cc2a456502e990593b58b641229d/Trajectory-WITSML-DC.txt)
Dataset record opendes:dataset--File.Generic:b5c31b0cacdf4238b13f098beb382694
Energyml_converter runID="runId": "f2051ce...Used this source data[Trajectory-WITSML-DC.txt](/uploads/9732cc2a456502e990593b58b641229d/Trajectory-WITSML-DC.txt)
Dataset record opendes:dataset--File.Generic:b5c31b0cacdf4238b13f098beb382694
Energyml_converter runID="runId": "f2051ce6-5bfc-4374-af89-0b2582fb4b9b",
Xcom summary shows record ID =
{'record_id': 'opendes:dataset--File.Generic:5d75967ee5384e6e8f7f5f38cb78ec3b:',
Using the above dataset record, we run Manifest Ingestion by reference.
"runId": "223ec4fc-b0c5-48df-bfaa-f4fe71047ed2",
Fails
Step provide_manifest_integrity_task
[{'id': 'opendes:work-product--WorkProduct:b102ceb0-41f2-452a-8fbf-762c5497d2d7', 'kind': 'osdu:wks:work-product--WorkProduct:1.0.0', 'reason': 'Missing parents: set()'}]
See JSON file created by EnergyML_converter[Generates-manifest.json](/uploads/76464c6f8b61446cffcd42c83f466e79/Generates-manifest.json)
There are many problems with the use of trailing colon.Valentin GauthierValentin Gauthierhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser-v2/-/issues/11M20/Azure/Preship - Wellbore data type - ID created with trailing colon2023-10-03T08:44:02ZDebasis ChatterjeeM20/Azure/Preship - Wellbore data type - ID created with trailing colonIt is not common convention for ID to have trailing colon.
Please consider revision to remove this.
['opendes:work-product--WorkProduct:fc6be8ae-83f0-4394-a164-7d8786595522',
'opendes:dataset--File.WITSML:411dddd1c4bf430ca45ee31ac155e9...It is not common convention for ID to have trailing colon.
Please consider revision to remove this.
['opendes:work-product--WorkProduct:fc6be8ae-83f0-4394-a164-7d8786595522',
'opendes:dataset--File.WITSML:411dddd1c4bf430ca45ee31ac155e9e6',
'opendes:master-data--Wellbore:3c321e10-d04a-4b6d-8bcd-5eae01cab52d:']
Note Wellbore record created with trailing colon in ID.Valentin GauthierValentin Gauthierhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser-v2/-/issues/10M20/Azure/Preship - Wellbore XML file - job fails with no meaningful error me...2023-09-29T07:22:21ZDebasis ChatterjeeM20/Azure/Preship - Wellbore XML file - job fails with no meaningful error message[Wellbore21-Peter-DC.xml](/uploads/ba28d197eb9a8604d3d7de934fb70a1a/Wellbore21-Peter-DC.xml)
Provided by @pgonzalez71 .
Tried to run the workflow using this data file.
Dataset record ID = opendes:dataset--File.Generic:315d6f9b8f7f4edeb...[Wellbore21-Peter-DC.xml](/uploads/ba28d197eb9a8604d3d7de934fb70a1a/Wellbore21-Peter-DC.xml)
Provided by @pgonzalez71 .
Tried to run the workflow using this data file.
Dataset record ID = opendes:dataset--File.Generic:315d6f9b8f7f4edebb029ace3a216f99
runID = "runId": "d42c3ad3-9b4e-458a-aba5-a4daaf08a34a",
Airflow log from step of content_loading.
[Airflow-log-content-loading.txt](/uploads/21c53290e8821686acee4e15dc3a4067/Airflow-log-content-loading.txt)Valentin GauthierValentin Gauthierhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/4EDS fetch - show in Airflow log quick test of a service in order to test succ...2022-10-07T11:35:09ZDebasis ChatterjeeEDS fetch - show in Airflow log quick test of a service in order to test success with authenticationAs per my recent discussion with @jeyakumar-jk -
This kind of test will iron out if user specification of source authentication information (Ex: client secret etc. via secret service) is causing any problem.
Execute search for example ...As per my recent discussion with @jeyakumar-jk -
This kind of test will iron out if user specification of source authentication information (Ex: client secret etc. via secret service) is causing any problem.
Execute search for example using common Reference Entity such as ExistenceKind.
Limit this kind of check for proper OSDU-compliant data source.
External non-OSDU sources may always not support query on Reference entity.https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/3EDS-Ingest - show in Airflow log exact body of query being executed to bring ...2022-11-23T11:29:57ZDebasis ChatterjeeEDS-Ingest - show in Airflow log exact body of query being executed to bring data from sourcePer recent discussion with @jeyakumar-jk
Such information will help user troubleshoot problems as needed.
Such as what query the eds_fetch-and-ingest is executing at external source?
(query constructed by using user provided **kind**...Per recent discussion with @jeyakumar-jk
Such information will help user troubleshoot problems as needed.
Such as what query the eds_fetch-and-ingest is executing at external source?
(query constructed by using user provided **kind** and **filter**)
Bonus - show one record id (put a limit of "1" in search body) that the search would return.
With that, the user gets peace of mind that input specifications as provided by him/her have no issues.M15 - Release 0.18Priyanka BhongadePriyanka Bhongadehttps://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/2EDS Preship test - fetch query does not respect wildcard syntax2023-08-23T16:21:29ZDebasis ChatterjeeEDS Preship test - fetch query does not respect wildcard syntaxAs discussed with @AshishSaxenaAccenture , we found that the following query failed to fetch single record (test case - fetch from AWS/Preship to Azure/Preship).
POST {{osduonaws_base_url}}/api/search/v2/query
```
{
"kind": "osdu:w...As discussed with @AshishSaxenaAccenture , we found that the following query failed to fetch single record (test case - fetch from AWS/Preship to Azure/Preship).
POST {{osduonaws_base_url}}/api/search/v2/query
```
{
"kind": "osdu:wks:master-data--Basin:1.0.0",
"query": "DCAWS*",
"returnedFields": ["id"]
}
```
After troubleshooting the Dev team found out that fetch will bring data if more precise search criteria is specified.
Hi Debasis,
You are giving the wrong Filter criteria.
We have updated the filter criteria in CSDJ as below:
` "Filter": "(data.BasinName: \"DCAWSBASIN1\")"`
And it successfully fetched one record.
run id : manual__2022-10-07T06:35:32+00:00
eds_ingest - Graph - Airflow (msft-osdu-test.org)
Regards,
AshishPriyanka BhongadePriyanka Bhongade