Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
  • S Storage
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 46
    • Issues 46
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 16
    • Merge requests 16
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Open Subsurface Data Universe SoftwareOpen Subsurface Data Universe Software
  • Platform
  • System
  • Storage
  • Merge requests
  • !539

ADR 92: Populate recordBlocks in record modification events

  • Review changes

  • Download
  • Email patches
  • Plain diff
Merged Abhishek Nanda requested to merge data-meta-compare into master Nov 04, 2022
  • Overview 5
  • Commits 11
  • Pipelines 16
  • Changes 11

Type of change

  • Bug Fix
  • Feature

Please provide link to gitlab issue or ADR(Architecture Decision Record)
ADR 92

Does this introduce a change in the core logic?

  • [YES/NO] No

Does this introduce a change in the cloud provider implementation, if so which cloud?

  • AWS
  • Azure
  • GCP
  • IBM (Added an additional field (hash) to the document db. Auto populated for all other providers.)

Does this introduce a breaking change?

  • [YES/NO] No

What is the current behavior?

  • skipdupes only works for changes in RecordData and ignores changes in RecordMetadata.
  • Record modification events do not contain info on which section was updated

What is the new/expected behavior?

  • skipdupes=true now handles for change in any of RecordData and RecordMetadata fields
  • This also implements ADR 92 which populates a recordBlocks field with the fields updated
  • skipdupes now uses the recordBlock field to check if any updates are made to the record

Have you added/updated Unit Tests and Integration Tests?

Unit Tests for all new files and new lines with 100% coverage has been added

Any other useful information

The hashes of data and meta field are now saved in metadata which is used to validate if any changes were made and prevents and extra call to the blob storage for each record. This is used both in skipdupes=true scenario as well as to populate the recordBlocks field. This brings around a 50% performance improvement in skipdupes=true scenarios

Edited Nov 07, 2022 by Abhishek Nanda
Assignee
Assign to
Reviewers
Request review from
Time tracking
Source branch: data-meta-compare