R3 Ingestion DAG fails to ingest official R3 manifests due to reference to ResourceSecurityClassification
When I execute attached body against the R3 ingestion dag, it fails due to a ResourceSecurityClassification record not being found. The official R3 manifests found here reference this in the ResourceSecurityClassification field: opendes:reference-data--ResourceSecurityClassification:RESTRICTED:
It seems as though the dag is trying to find a record by that id in the system. The problem is that this id can't be used to make any record, it throws a 400 on storage. So the dag logs this as a warning:
And then doesn't actually make any records even though it finishes as success.
Either the official manifests need to be updated to have an actual record id there or the dag needs to be updated to not look for the resource security classification record using that field as an id.
Specifically, this logic happens on line 181 of validate_referential_integrity.py
:
Other than this problem, I was able to successfully ingest manifests with this dag on AWS.