Latency vs throughput

A dashboard says the system handled more requests last week than it ever has, and in the same week support hears that it feels slower. Both readings can be correct, because they are counting two different things. The dashboard is almost certainly counting throughput — how much work gets through in a given stretch of time. The complaint is almost always about latency — how long one particular thing, the one the person is staring at, took to come back. A system can get busier and slower for the same request at the very same time, and the first useful step is working out which of the two numbers the user is actually reporting.

Two numbers that move independently#

Picture a service handling requests through a single queue and a fixed pool of workers. On a busy stretch the queue clears more requests than it did an hour before — throughput is climbing, and by that number alone the system looks healthier than ever. Ask any one request how long it personally waited, though, and the picture flips: every request now sits behind more work than it used to, so the latency of any single request is worse than when the queue was short. Same system, same stretch of time, both numbers correct and pointing in opposite directions, because one counts finished work and the other counts time spent by one thing.

Bandwidth gets pulled into this argument constantly, and it is neither of the two. Bandwidth is a capacity — the most a link or a system could carry in a given stretch of time — not a measurement of what actually happened. A connection can carry ample bandwidth while its measured throughput sits well under that ceiling, because congestion, contention, or a slow consumer on the other end can hold actual delivery below what the link is capable of. Latency and throughput describe what happened; bandwidth describes what was available to happen.

The figure that quietly goes missing in all of this is the average. A mean latency taken across every request blends the ordinary case and the disaster case into one number, and the disaster case is the one a person actually notices and reports. A handful of requests that took far longer than the rest barely move the mean, but they are exactly the ones somebody remembers — which is why a serious report of latency is drawn from the slow end of the distribution rather than from the average across all of it. An improving mean and a growing pile of complaints are not a contradiction; they are two different slices of the same data, and only one of them is what the frustrated user actually lived through.

The trade you make on purpose#

Batching is the clearest place this trade gets made on purpose rather than by accident. Handling items one at a time pays a fixed cost — a round trip, a lock, a flush to disk — separately for each one; handling the same items as one batch pays that fixed cost once and spreads it across the group, so throughput rises because the system gets through more work for the same fixed overhead. The cost lands on latency: the first item placed into a batch does not finish until the batch fills and the whole group is processed together, so that item now waits for company that has not arrived yet.

The same shape reappears anywhere work is allowed to pile up before it is acted on. A queue that absorbs a burst of arrivals and works through it steadily, instead of falling over, is smoothing throughput at the cost of making every arrival wait its turn rather than being handled the moment it lands. A connection pool that holds a caller until a slot frees up is doing the identical thing at a different layer — protecting the pool’s own throughput by making the caller’s latency someone else’s problem for a moment. Any buffer, anywhere in a system, is this same choice wearing a different name.

None of that is a mistake by itself; it is a design decision with a real question attached: whose waiting are you spending? A nightly job that nobody is watching can absorb a great deal of waiting and nobody notices, because no person is sitting in front of it counting time. A page somebody is looking at right now has a far stricter budget for the same waiting, because the person doing the waiting is the one who feels every bit of it. The trade is identical in both cases; only the answer to whose waiting it is changes what the right amount of it looks like.

The same choice about who pays and when shows up a layer down, inside a single storage engine rather than across a whole system’s request traffic. The trade between a B-tree and an LSM-tree is not framed in these two words at all — its own case is that a B-tree pays its cost at write time and an LSM-tree defers that cost to read time — but the shape underneath is the same one just named again: pay now in a way somebody waits for, or defer the cost onto whoever asks next.

Optimising the wrong number#

The failure shape is optimising the average and making the actual experience worse. A team notices throughput straining under load and reaches for one of the tools the previous section named — a bigger batch, a deeper queue, more work allowed in flight at once — and ships it with a graph showing the summary number moving the right way. What the graph does not show is the request that now sits behind more accumulated work than before: the slowest requests get slower even as the number at the top of the dashboard improves, because more work in flight means more for any given request to wait behind.

The tell is that the summary metric keeps improving while complaints do not fall, and the two are not disagreeing with each other — they are measuring different things and both are telling the truth. A throughput figure going up says more work is getting done. It says nothing about whether the particular thing a person is waiting on is any faster, and past a certain point the two move in opposite directions on purpose, because the same change that improved one is now the thing costing the other.

None of this makes throughput work a mistake in general — for a nightly job with nobody watching it, a bigger batch and a fuller queue are exactly the right call, for exactly the reason the previous section gave. What turns the same move into a failure is applying it to a request where a person is waiting on the other end, and calling the improved graph a win without checking who is still waiting behind it. A network’s own delay adds a third complication to both of these once more than one machine is involved — the wait now includes hops that neither number alone accounts for — and that is the ground the distributed-systems path is meant to eventually reach; it carries no published articles yet, only the roadmap for them.