Skip to content
GitLab
  • Menu
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in / Register
  • I Indexer
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 30
    • Issues 30
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 12
    • Merge requests 12
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages & Registries
    • Packages & Registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Open Subsurface Data Universe Software
  • Platform
  • System
  • Indexer
  • Merge requests
  • !111

Handling the storage api calls correctly

  • Review changes

  • Download
  • Email patches
  • Plain diff
Closed Aman Verma requested to merge users/amaverma/StorageCallsInALoop into master Mar 04, 2021
  • Overview 7
  • Commits 8
  • Pipelines 18
  • Changes 1

Problem

The reindexer service tries to call storage service for fetching records of a given kind. But the catch is that these calls are made in batches. In case the number of records for a given size are more than batch size, there is no handling in place to read rest of the records. Hence each time we call reindex api for a certain kind Only Tope 20 records are getting reindexed

FIX

Putting the storage service related calls in a while loop.

Cursor = null
1. call the storage service with batch size = 20 and batch size
2. If no records are found for this kind then break
3. If there are records in current batch then put those records in indexer queue
4. If this was the last batch- terminate, else update the cursor to location of next batch and repeat from step 1

Tested in dev environment for one kind

cc: @manishk , @kiveerap

Assignee
Assign to
Reviewer
Request review from
Time tracking
Source branch: users/amaverma/StorageCallsInALoop