Latency vs throughput

A dashboard says the system handled more requests last week than it ever has, and in the same week support hears that it feels slower. Both readings can be correct, because they are counting two different things. The dashboard is almost certainly counting throughput — how much work gets through in a given stretch of time. The complaint is almost always about latency — how long one particular thing, the one the person is staring at, took to come back. A system can get busier and slower for the same request at the very same time, and the first useful step is working out which of the two numbers the user is actually reporting.

Two numbers that move independently#

Picture a service handling requests through a single queue and a fixed pool of workers. On a busy stretch the queue clears more requests than it did an hour before — throughput is climbing, and by that number alone the system looks healthier than ever. Ask any one request how long it personally waited, though, and the picture flips: every request now sits behind more work than it used to, so the latency of any single request is worse than when the queue was short. Same system, same stretch of time, both numbers correct and pointing in opposite directions, because one counts finished work and the other counts time spent by one thing.

Put units on that distinction. In an illustrative, otherwise idle single-worker service, each request needs 10 milliseconds of processing, so the worker can finish at most 100 requests per second. If ten requests arrive together and run first-in, first-out, the first finishes after about 10 milliseconds and the tenth after about 100 milliseconds. The worker completed ten requests in 100 milliseconds, yet the requests had very different individual latencies. The numbers describe this simple model, not a measured production service.

Bandwidth gets pulled into this argument constantly, and it is neither of the two. Bandwidth is a capacity — the most a link or a system could carry in a given stretch of time — not a measurement of what actually happened. A connection can carry ample bandwidth while its measured throughput sits well under that ceiling, because network congestion, contention, or a slow consumer on the other end can hold actual delivery below what the link is capable of. Latency and throughput describe what happened; bandwidth describes what was available to happen.

For a network example, an 80-megabit-per-second link has a theoretical payload ceiling of 10 megabytes per second before protocol overhead. If the application actually delivers 6 megabytes per second, its observed byte throughput is 48 megabits per second; the link’s nominal bandwidth is still 80 megabits per second. Neither figure tells you whether one request waited 20 or 500 milliseconds. Keep request throughput (requests per second) separate from byte throughput (bytes per second) when comparing dashboards.

A mean can move substantially when even one request is extremely slow; it also cannot tell you how many requests were slow. Percentiles describe positions in a distribution, with the meaning depending on the time window, sample size and calculation convention. Compare median, tail percentiles, mean, errors and throughput rather than assuming one number captures every user’s experience.

Move one outlier and compare the summaries

Twenty completed requests: 19 take 10 ms; one is slower. Predict which summaries change when that last request gets slower.

Slow request latency (ms)

At 1000 ms: mean = 59.5 ms, median = 10 ms, nearest-rank p95 = 10 ms, p99 = 1000 ms. An extreme outlier can move the mean substantially and still be absent from p95.

A fixed 20-request sample and nearest-rank percentiles, not a production histogram or throughput calculation. Time window, sample size and percentile convention matter. Observe failures and queue age too; no summary describes every request.

The trade you make on purpose#

Batching is the clearest place this trade gets made on purpose rather than by accident. Handling items one at a time pays a fixed cost — a round trip, a lock, a flush to disk — separately for each one; handling the same items as one batch pays that fixed cost once and spreads it across the group, so throughput rises because the system gets through more work for the same fixed overhead. The cost lands on latency: the first item placed into a batch does not finish until the batch fills and the whole group is processed together, so that item now waits for company that has not arrived yet.

The same shape reappears anywhere work is allowed to pile up before it is acted on. A queue that absorbs a burst of arrivals and works through it steadily, instead of falling over, is smoothing throughput at the cost of making every arrival wait its turn rather than being handled the moment it lands. A connection pool that holds a caller until a slot frees up is doing the identical thing at a different layer — protecting the pool’s own throughput by making the caller’s latency someone else’s problem for a moment. Any buffer, anywhere in a system, is this same choice wearing a different name.

A queue cannot absorb overload forever. In the single-worker example, a sustained arrival rate of 120 requests per second against a 100-request-per-second processing ceiling adds at least 20 waiting requests each second, assuming no timeouts or drops. Completed throughput cannot exceed the worker’s capacity, while queue age and the slow end of the latency distribution keep rising. This is saturation, not a universal inverse formula between latency and throughput. Headroom helps absorb bursts; a bounded queue, backpressure or deliberate load shedding prevents unbounded waiting when demand stays above capacity.

None of that is a mistake by itself; it is a design decision with a real question attached: whose waiting are you spending? A nightly job that nobody is watching can absorb a great deal of waiting and nobody notices, because no person is sitting in front of it counting time. A page somebody is looking at right now has a far stricter budget for the same waiting, because the person doing the waiting is the one who feels every bit of it. The trade is identical in both cases; only the answer to whose waiting it is changes what the right amount of it looks like.

The same choice about who pays and when shows up a layer down, inside a single storage engine rather than across a whole system’s request traffic. The trade between a B-tree and an LSM-tree is not framed in these two words at all — its own case includes immediate page updates versus buffered writes and later compaction, with both read and write costs — but the shape underneath is the same one just named again: pay now in a way somebody waits for, or defer the cost onto whoever asks next.

Optimising the wrong number#

The failure shape is optimising the average and making the actual experience worse. A team notices throughput straining under load and reaches for one of the tools the previous section named — a bigger batch, a deeper queue, more work allowed in flight at once — and ships it with a graph showing the summary number moving the right way. What the graph does not show is the request that now sits behind more accumulated work than before: the slowest requests get slower even as the number at the top of the dashboard improves, because more work in flight means more for any given request to wait behind.

The tell is that the summary metric keeps improving while complaints do not fall, and the two are not disagreeing with each other — they are measuring different things and both are telling the truth. A throughput figure going up says more work is getting done. It says nothing about whether the particular thing a person is waiting on is any faster, and past a certain point the two move in opposite directions on purpose, because the same change that improved one is now the thing costing the other.

For an interactive endpoint, measure end-to-end latency in milliseconds at p50 and p95 or p99, queue age, completion rate and errors together. Find the saturated resource before adding concurrency: more simultaneous requests can raise throughput until they contend on a shared CPU, database or link, after which it can enlarge the queue instead. Cache repeated reads when the freshness contract permits, add capacity at the actual bottleneck, and reject excess work rather than promising users an answer after their deadline. For a background job, measure items per second and the job’s completion deadline; larger batches and a bounded queue may be worthwhile even if each item waits longer.

None of this makes throughput work a mistake in general — for a nightly job with nobody watching it, a bigger batch and a bounded queue can be the right call. What turns the same move into a failure is applying it to a request where a person is waiting on the other end, and calling the improved graph a win without checking who is still waiting behind it. Once more than one machine is involved, network delay adds to processing and queueing time; the distributed-systems path covers related trade-offs such as moving load across nodes.

Questions people also ask

5 QUESTIONS

▸ Can latency and throughput both get worse at once?

Yes. They trade against each other only when one is deliberately shifted toward the other — a bigger batch, a fuller queue. Under genuine resource exhaustion, such as a shared disk or CPU running out of headroom, every request slows down and fewer finish per stretch of time together, because contention costs both at once rather than trading one for the other.

▸ Is bandwidth the same as throughput?

No. Bandwidth is a capacity — the most a link or a system could carry in a given stretch of time. Throughput is what actually got carried. A connection with generous bandwidth can still show modest throughput if congestion, a slow consumer, or contention elsewhere holds real delivery well under the ceiling the link is capable of.

▸ Why quote a percentile instead of an average?

An average blends the ordinary case and the disaster case into one number, and the disaster case is what a person actually notices. A sufficiently extreme outlier can move the mean substantially. Percentiles help describe the distribution, but a chosen percentile can also exclude a rarer slow request. Report the window, count and convention along with typical and tail latency.

▸ Does adding servers reduce latency?

Not directly. More servers usually raise throughput, because more work can run in parallel, but a single request still passes through the same steps and still waits behind whatever else competes for the same downstream resource. Latency falls only if the added capacity actually shortens that particular request's own wait, which is a separate question from how much total work the system can now handle.

▸ Why does batching make things slower for users?

Batching pays a fixed cost once instead of once per item, which is why it raises throughput. The item placed into a batch first has to wait for the rest of the batch to fill before the whole group gets processed together, so its own latency grows even though the system as a whole is getting more done. The waiting moved; it did not disappear.