AZURE M22 - Ingestion Services: manifest processing with providing integrity
I tried to ingest a manifest (attached file test-resqml-geosiris-azure.json, I faced an issue during the "manifest processing with providing integrity" operator task.
The reason of a skipped ids is : 'reason': "'recordIds'"
. The others 'Missing parents' reasons are related to the first not ingested element (opendes:work-product-component--LocalBoundaryFeature:5fa99eb4-b11f-4f08-b1be-2d64ff14286f), because other WPC refer to it.
By the way I think it is strange the 'missing parent' makes the ingestion fail because a WPC with this id is allready present in the catalog
Link of the airflow dag run xcom values : link
Hypotheses:
-
The input manifest has other errors that makes the ingestion fail (and maybe the 'reason' is not too specific to understand why) :
- The validation in the previous operator worked : I can't see anything specific in the logs.
- I tried to change all UUIDs (present in the WPC "id") to see if the problem is really on the "id" value and nothing else. I then produced this new manifest :test-resqml-geosiris-azure-reidfied.json. And the ingestion worked.
- Conclusion : the problem is really on these specific ids (the generated "datasets" are never skipped because their id are generated randomly after a push on the DatasetService)
-
When a recordId allready exists in the catalog, the ingestion fail
- I tested to ingest twice an other manifest with WITSML data test-witsml-geosiris.json). And it worked.
- I tried to modify in the previously ingested WITSML manifest the version number, tags to see if the ingestion did not failed because the input was exactly the same. But even with these small modification it worked.
- Conclusion : even if an entity with the same "id" is present in the catalog, the ingestion works.
The only difference I see with my "double ingestion" (that worked) and the first ingestion (that failed), is that in the catalog the current version of the WPC (not mine) contains "DDMS" attributes.
Is there any chance that the DDMS attributes give to WPC a security to avoid WPC replacement and keep the integrity of the DDMS ?
Help :
-
Get the current version of the WPC that makes mine fail to be ingested :
POSTMAN GET request : https://{{STORAGE_HOST}}/records/opendes:work-product-component--LocalBoundaryFeature:5fa99eb4-b11f-4f08-b1be-2d64ff14286f