Skip to content

Refactor AWS implementation of Indexer Queue

Derek Hudson requested to merge aws-refactor into master

Refactor of AWS implementation of Indexer Queue. Instead of waiting for a full batch of up to 10000 messages, then processing the up to the 10000 messages, then deleting (or changing visibility or sending the message to a dead letter queue) that batch, this change gets messages in up to batches of 10, sends them to the service-local queue that is then pulled from the worker thread and then sent to a delete (or change visibility or dead letter queue) message batch worker that completes the process. The overall throughput has improved significantly (able to consistently process >200 messages per minute) without an increase in thread workers.

Edited by Derek Hudson

Merge request reports

Loading