GONRG-2696: Manifest integrity batch search
Type of change
Does this introduce a change in the core logic?
Does this introduce a change in the cloud provider implementation, if so which cloud?
Reimplement logic of Manifest Integrity Check.
Before this MR, each entity of the Manifest was traversed during Manifest Integrity check. During this traversing, references to other entities (both in the Manifest or on OSDU) were collected and if they weren't present in Manifest, they were searched in Search service. And if there were missing entities on OSDU, then the entity containing these missing references was excluded from the manifest. This sequence of actions were repeating for each entity in the Manifest.
Now, the process of Integrity Check has following steps:
- Traverse the Manifest and create the dependency graph with parent-child relations between entities. Referrer is a child, referent is a parent.
- If there are references to
entitiesnot in the Manifest, mark such
- Try to find all
externalentities in Search Service splitting the list of external entities into chunks.
- If there are missing entities (records) on OSDU, mark them and their children (referrers) as invalid and exclude them from the manifest.