Rate limiting

Rate limiting looks like one decision — how many requests per unit of time — and is really three, only one of which any specification has an opinion about. The protocol tells you how to refuse a request. Everything that determines whether your limit actually holds is left open on purpose, and the defaults people reach for have a well-known hole in them.

What the specification actually fixes#

RFC 6585 introduces 429 Too Many Requests, and the definition is exactly as narrow as it needs to be: the status code indicates that the user has sent too many requests in a given amount of time. Alongside it the specification asks that the response explain the condition, and allows a Retry-After header saying how long to wait. That header is worth treating as the point of the whole exchange rather than a nicety — it is the only thing in the response that tells a well-behaved client what to do next, and without it every client is left guessing at a backoff.

Then the specification stops, and says so. It does not define how the origin server identifies the user, nor how it counts requests. That is the right call — how to count is a question about your traffic, not about HTTP — but it means the interesting half of any rate limiter lives in territory no standard covers. Two services can both answer 429 correctly and enforce wildly different things.

The window decides the burst#

The simplest counter is a fixed window: count requests per client, reset on the hour. It is easy to reason about, cheap to store, and it permits twice the stated limit whenever a client wants it. Spend the full allowance in the last minute before the reset, then the fresh allowance in the first minute after, and a limit of a hundred an hour has just delivered two hundred requests in roughly two minutes — while the client stayed inside the published rule the whole time.

The alternatives are all ways of refusing to have a boundary. A sliding window counts over the trailing period rather than the current bucket, so there is no moment at which the count is forgiven all at once. A token bucket takes it further and separates two things the fixed window conflates: the long-run rate, set by how fast tokens are added, and the burst, set by how many the bucket can hold. That separation is usually what you actually wanted — most services are happy to absorb a short spike and only care about the sustained rate, and the fixed window gives you no way to say that.

When a 429 gets cached#

The detail most implementations miss sits in one line of RFC 6585: responses with the 429 status code must not be stored by a cache. It is easy to see why the rule exists once you picture the alternative — a cached 429 keeps refusing a client long after the limit has cleared, and it can refuse clients that were never over it. Anything that stores responses by status code without exception is capable of turning a momentary limit into a lasting outage.

The second is choosing the wrong thing to count against. Limiting per IP address gathers an entire office or mobile network behind one counter, so the limit that stops one abusive client also stops a hundred innocent ones. Limiting per API key is fairer and does nothing about an unauthenticated endpoint. Whatever you pick has to be visible where the limit is enforced, which is often a layer that sees the connection but not the caller. There is no correct answer, which is precisely why the specification declined to give one — but the choice needs making deliberately, because it decides who gets hurt when the limit does its job.

The third is what happens after the refusal. A client that retries immediately turns one rejected request into many, and a fleet of them doing it together turns a limit into an attack on your own service. This is where the limiter and the client’s retry policy have to be designed as one thing — with a backoff that spreads retries out, and an endpoint that treats the eventual successful retry as routine rather than as a second request.