Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Register
  • Sign in
  • I Indexer
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
    • Locked files
  • Issues 25
    • Issues 25
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 12
    • Merge requests 12
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Open Subsurface Data Universe SoftwareOpen Subsurface Data Universe Software
  • Platform
  • System
  • Indexer
  • Merge requests
  • !112

Moving the while loop to iterate over storage records in azure code

  • Review changes

  • Download
  • Email patches
  • Plain diff
Merged Aman Verma requested to merge users/amaverma/makeWorkerAsync into master Mar 05, 2021
  • Overview 0
  • Commits 6
  • Pipelines 5
  • Changes 1

Problem

The reindexer service tries to call storage service for fetching records of a given kind. But the catch is that these calls are made in batches. In case the number of records for a given size are more than batch size, there is no handling in place to read rest of the records. Hence each time we call reindex api for a certain kind Only Tope 20 records are getting reindexed

FIX

Putting the storage service related calls in a while loop.

Cursor = null
1. call the storage service with batch size = 20 and batch size
2. If no records are found for this kind then break
3. If there are records in current batch then put those records in indexer queue
4. If this was the last batch- terminate, else update the cursor to location of next batch and repeat from step 1

Tested in dev environment for one kind

cc: @manishk , @kiveerap

linked with !111 (closed)

Assignee
Assign to
Reviewers
Request review from
Time tracking
Source branch: users/amaverma/makeWorkerAsync