SPIKE: Data - Investigate options for incremental updates

Cache updates should take into account what records have been loaded since the last update, and then attempt to add only those new records instead of doing a complete destroy-rebuild of the Ignite cache.

Important details:

• Need to track lastUpdatedTime within Transformer/Ignite
• What property on a Search API record dictates the last time the record was modified or added?
• Need to enforce that new records abide by the schema currently set for the Ignite Cache. New records that break the Ignite schema will not be added. Schema cannot be modified once it is in place.
  • Possible solution: Stick with same kind version for incremental updates to avoid schema conflicts.
• Cron Jobs can be trigger for incremental updates
• Often, existing records within OSDU may have their attributes adjusted.
  • On incremental update, we should track when records were last updated and update all records that have been updated since the last GCZ update.