Skip to content

Fix data inconsistency issue between metadata storage and blob storage

Alok Joshi requested to merge fix_data_inconsistency into master

Issue

We've observed data inconsistency issues for some storage record objects where the latest version of record (available in metadata storage - i.e. cosmosDb) is soft deleted from blob storage (i.e. Azure Storage). This is causing exception for APIs like GET /api/storage/v2/records/<id> and POST /api/storage/v2/query/records:batch

Solution

We propose to fix this data inconsistency by two changes:

  • Clean up datastore as well, when cleaning up cloud storage, in case of commitBatch method exception in PersistenceServiceImpl (core logic change)
  • For existing records having this issue: try restoring the data from blob storage. We are attempting to restore the latest record version (recorded in cosmosDb) from blob storage if its not found

Merge request reports