Enhance Reference Data Mapping as per XOM suggestion[Techdebt]

To improve the performance of reference mapping and provide a more simplified ExternalReferenceValueMapping record

  • To change the logic to get reference data in demand rather than load all into memory initially
  • Scope will hold the CSRE id
  • SimpleMap.ReferenceValueID will hold the target reference mapping value
  • HasGlobalScope will be default considered as false, which means the reference data mapping is specific to a particular data provider
  • HasGlobalScope = true, which means the Scope will not be considered and it NOT specific to a particular data provider
  • New Schema ExternalReferenceValueMappingSet for new nested structure[require feasibility study in M26]
  • Deployment of new Schema
  • Implementation based on the new schema and use old schema for backward compatibility

MR for HasGlobalScope= true: osdu/platform/data-flow/ingestion/osdu-airflow-lib!355 (merged)

Edited by Jeyakumar Devarajulu