Exponential backoff

A retry is the most reasonable-looking line of code in any client. The call failed, the failure was probably transient, so try again — and every part of that reasoning is sound in isolation. What makes retries dangerous is that they are correlated: they fire when something is wrong, which means they arrive together, at the worst possible moment, from everywhere at once.

Retries multiply, they do not add#

Google’s SRE book walks the loop directly. A backend reaches capacity and begins rejecting requests; the rejections prompt the frontend to retry; the retries are additional load. The volume grows — its own numbers are 100 queries per second of retries in the first second, then 200, then 300. The service is being asked to do more work as a direct consequence of being unable to do the original work, and nothing in that loop is self-limiting.

The part people underestimate is what happens when retries exist at more than one level, which they usually do because different teams added them at different times. The SRE book gives the arithmetic: with a client, a frontend and a backend each retrying three times, a single user action can produce 64 attempts at the lowest layer. Three independently sensible policies compose into one that nobody would have written down.

That multiplication is the argument for deciding where retries live rather than letting each layer decide for itself. A layer that retries needs to know whether the layer beneath it already did.

Backoff is half of it#

Exponential backoff addresses the first problem: rather than retrying immediately and repeatedly, each attempt waits longer than the last. One client’s attempts spread out, the pressure it applies falls away quickly, and a service that needs a few seconds to recover gets them.

What backoff does not fix is that the clients are synchronised. A thousand clients failing at the same instant, all doubling their waits, all come back at the same instant — a second later, then two, then four. The load is spread over time and remains concentrated in spikes, and each spike is capable of knocking over a service that had just started to recover. Which is why the SRE book’s guidance is to always use randomized exponential backoff. The randomisation is not a refinement on the backoff; it is the half that breaks the correlation between clients, and backoff without it produces a slower thundering herd rather than no herd.

Which raises the question the advice usually leaves hanging: randomise how? AWS’s published comparison names three variants and simulates them against each other, and they differ in what they keep of the backoff. Full jitter keeps none of it: the sleep is a random value drawn between zero and the computed backoff, written in the article as sleep = random_between(0, min(cap, base * 2 ** attempt)). Equal jitter keeps half — half the computed backoff, held fixed, plus a random amount up to the other half — on the reasoning that a client should not be able to retry almost immediately after a long wait. Decorrelated jitter abandons the attempt counter and derives each wait from the previous one instead, growing the range as it goes rather than following a fixed curve.

The simulation is what makes the choice tractable, and it does not crown a single winner. Full and equal jitter came out close on total work, with decorrelated doing more of it; equal jitter was the slowest to complete, and full jitter did less work than decorrelated at a small cost in time. All three cut the work sharply against backoff with no jitter at all, which is the finding that matters: the gap between some randomisation and none is far larger than the gap between the three ways of doing it. Pick full jitter unless you have a reason not to, and treat the choice between them as a tuning question rather than the decision.

What a budget is for#

Backoff and jitter shape retries; they do not cap them. The SRE book adds two limits that do. A per-request cap stops one call retrying indefinitely, and a process-wide retry budget — its example is allowing only 60 retries per minute in a process — bounds the total regardless of how many individual calls are failing. The budget is the one that saves you during a broad outage, because that is exactly when every request is failing and every per-request cap is being honoured.

The cheapest protection is also the most commonly skipped: not retrying what cannot succeed. A malformed request, a rejected credential, a resource that does not exist — none of these change between attempts, so retrying spends capacity to relearn a known fact. Distinguishing retriable errors from permanent ones costs a branch and removes a whole category of wasted load.

Backoff, jitter and a budget all pace retries. None of them stops one, and there is a case where stopping is the only correct move: the dependency is not briefly overloaded, it is down, and every request you send will fail after burning a timeout on the way. A circuit breaker is the mechanism for that, and it is a state machine rather than a delay. Resilience4j’s implementation is a fair reference for the shape — it names three normal states, CLOSED, OPEN and HALF_OPEN. Closed is ordinary operation. Once the failure rate crosses a configured threshold, and only after a minimum number of calls have been recorded, it opens, and an open breaker stops calling the dependency at all: requests are rejected immediately rather than sent and timed out.

The interesting state is the third one, because it is where a breaker could re-create the very problem this page is about. After a wait, the breaker moves to half-open and permits a configured number of calls through — ten by default in that implementation — to find out whether the dependency has recovered, rejecting everything else until those permitted calls have finished. If their failure rate is still above the threshold it returns to open; if it is below, it closes. That deliberate cap is the difference between probing a recovering service and hitting it with the whole fleet the moment the timer expires — which is the thundering herd from the previous section, arriving through the mechanism that was supposed to prevent it.

Two things elsewhere in the system finish the design. The server needs a way to say not now and be understood, which is what a rate limiter’s response is for — and a client that honours the wait it is given beats any backoff it invents locally. And because a retry that succeeds may be a duplicate of one that also succeeded, the receiving end has to be built so a repeated request is a routine case rather than a second one. Retry policy is a property of the pair, not of the caller.