Fetch-and-Ingest - logic for handling company name (Organisation Master data)
Provider sends Master data SeismicAcquisitionSurvey. One of the field uses Organisation – record ID=BP (meaning Owner/Operator = BP)
Operator who is receiving data from Provider may have this company recorded in Organisation. But they may have synthetic ID. And name = BP may be a field for the record.
Such as from https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/Examples/master-data/Organisation.1.1.0.json
"OrganisationID": "Example External Organisation Identifier",
"OrganisationName": "Example OrganisationName",
The question is – do we entertain this kind of match/checking during ingest of fetch and ingest? If yes, this should be considered a priority. What do you think?
Proposed solution - fetch-and-ingest may search for value "BP" directly in ID, next search for "BP" in OrganisationName from Organisation. If it finds match, then it should alter incoming JSON payload and provide suitable ID. So that incoming data can be ingested successfully.
If it finds multiple matches, then the first record may be used.