add backoff on search and storage client

Merged Yannick requested to merge feature-backoff into master

Adding backoff strat to retry on some specific error on search and storage client. It will retry on few cases for now, already identified (basically connection failure). It used exponential wait strategy. The number of max_tries and max_wait_times are configurable through env variable. Default are respectively 4 and 5 seconds. The factor and base wait are fixed and set to 1 and 0.5 second.

There's one handler so far to trace the error in case of backoff.

Note: this MR also contains a trace added around the 'write bulk', used during investigation which appeared quite useful.

Edited by Yannick

Merge request reports