M25 EDS - Mapped Master data - Complex Mapping
As per provided example for complex mapping on: https://osdu.pages.opengroup.org/platform/data-flow/ingestion/osdu-airflow-lib/eds-configuration-guide/#mapped-master-data-complex-mapping
The example for ParentMatchingRuleSets
"Conditions": [
{
"SourceProperty": "data.GeoContexts.GeoTypeID: {{operator_partition}}:reference-data--GeoPoliticalEntityType:Country",
"TargetKind": "master-data--GeoPoliticalEntity:1.1.0",
"TargetProperty": "data.GeoPoliticalEntityName",
"ReplaceProperty": "data.GeoContexts.GeoPoliticalEntityID",
}
]
Notice the space in SourceProperty after GeoTypeID. This example does not work and causes DAG to crash. Removing the space leads to successful run.
The exeption is thrown in osdu_airflow/eds/eds_ingest/utilities/helper_value_utility.py", line 77:
extracted_data_type = master_data.split(":")[1]
Here master_data is a list instead of str. Instead if the space in ParentMatchingRuleSets is removed, everything works as expected. This either requires update in documentation or change in regular expressions that parse SourceProperty for complex mapping.
{
"SourceProperty": "data.GeoContexts.GeoTypeID:osdu:reference-data--GeoPoliticalEntityType:Country",
"TargetKind": "master-data--GeoPoliticalEntity:1.0.0",
"TargetProperty": "data.GeoPoliticalEntityName",
"ReplaceProperty": "data.GeoContexts.GeoPoliticalEntityID",
}
Edited by Arpit Singh