Load balancers, concretely
See how round robin, least connections and a hash actually differ, and which of them stops working the moment a server keeps anything in its own memory.
The interesting part is not the spreading, it is what the chosen server is assumed to know. Round robin, least connections and hashing pick differently, and each one quietly assumes something about where session state lives.
Think of a supermarket with somebody directing shoppers to tills. Sending each person to the next till in turn works fine until one shopper has a half-packed trolley waiting at till three, and gets sent to till five.
Two servers sit behind something that spreads requests between them, and the question that actually matters is not which algorithm looks fastest on paper — it is whether the two servers are truly interchangeable. Get that part wrong and no algorithm saves you: users start getting logged out at random, nothing in the logs names a cause, and the balancer gets blamed for a decision it never made. This is what the three common ways of picking a server actually do, what each one assumes about the servers behind it, and the one assumption that breaks all three the same way.
Three ways to pick, and what each assumes#
| Algorithm | How it chooses | Assumes about the servers | Assumes about the request | Where it misbehaves |
|---|---|---|---|---|
| Round robin | Next server in a fixed rotation | Roughly equal capacity | Roughly equal cost | An expensive request stalls a server that then keeps its turn |
| Least connections | Server with the fewest open connections | Open connections track load | Cost is not known in advance | A server that fails fast holds few connections and attracts more |
| Hash of a client attribute | A hash of something about the client, e.g. an address | The pool stays stable | The same client should land in the same place | Changing the pool reshuffles the mapping for a share of every client |
| Random | A server picked at random each time | Nothing about current load | Nothing about cost | Short runs of bad luck send several expensive requests to one server in a row |
Round robin assigns successive requests in rotation without consulting their cost. A slow backend can therefore keep receiving its share while a faster backend has spare capacity. Independent backends do not inherently wait for each other: the problem is uneven queued work, not a rule that a fast server must idle behind a slow request.
Least connections measures the exact thing round robin has no view of: it sends each new request to whichever server currently has the fewest connections open, which is a reasonable stand-in for how busy that server is. Its failure mode is the mirror image of round robin’s, and it is the sharper one to watch for. A server with a broken dependency that answers every request with an immediate error looks, by this measure, like the least busy server in the pool — its connections close fast because it isn’t doing any real work, and least connections keeps sending it more traffic precisely because it looks free. A server failing quickly can pull load toward itself instead of away from it.
A stable client attribute can select the same server while the pool and mapping stay stable. A cookie may instead directly record a selected backend; it need not be a hash result. Adding or removing servers can remap clients, with the extent depending on ordinary modulo hashing, consistent hashing, weights and the product’s failure rules. Affinity does not itself replicate a session or preserve it through a server restart.
One distinction underlies why a hash can even be computed on a client attribute in the first place. Choosing a server from connection-level information — an address and a port, before anything about the request has been read — is a different amount of information from choosing after the request itself has been parsed. A hash on a header, a cookie or a path needs the second kind of decision; a hash on the connection’s own address does not. Which layer a balancer decides at is not a detail — it is the difference between having that information available to hash on at all and not.
Route a session to a different server
Login created session S on server A. The next request goes to B. Predict whether the routing decision can supply missing state.
B cannot find S in its own memory. Returning to A would work while A still has S; shared storage would let either server find it.
Two available servers, one session, successful store reads. This models state location, not a hashing implementation, security policy, store outage or performance. Stickiness cannot recover memory lost when A restarts.
What the health check actually proves#
A health check proves exactly what it asks and nothing more. An endpoint that reports healthy because the process is running and able to answer HTTP at all says nothing about whether that process can reach the database, the cache or whatever else it actually needs to do its job. A server with a dead dependency stays in the pool under a check like that, and answers every request it receives with an error — fast, because failing needs no real work, which under least connections is exactly the condition that draws more traffic toward it rather than away.
The fix is naming two different questions rather than asking one and hoping it covers both. A liveness check asks whether the process is alive at all — whether it should be restarted if it is stuck or has crashed. A readiness check asks something narrower and more useful to a balancer: can this specific instance serve a request right now. A pool that only ever asks the first question will happily route traffic to an instance that is alive, responding, and unable to do anything useful with what it receives — which is precisely the gap a check built only to catch a crashed process leaves open.
The honest complication runs the other way too. A readiness check that reaches all the way through to every dependency the server touches is more accurate about that one instance, but it also means a dependency shared by the whole pool — one thing every instance calls out to — can take every instance out of rotation at once if it wobbles. A shallow check misses a real problem; too deep a check turns one shared dependency’s bad moment into every server failing its check simultaneously, which is a total outage standing in for what was actually a partial one.
Logged out after every deploy#
The failure that actually reaches production starts with session state kept in one server’s own memory. It works, for a while — either every request from a given user happens to keep landing on the same instance by chance, or stickiness is switched on and nobody thought hard about what it was quietly holding together. Then a deploy rolls through, or the pool scales up, or a single instance restarts, and the mapping that stickiness depended on moves. Users are logged out at random. There is no error anywhere that names the cause, because nothing failed — the request that lost its session was answered correctly by a server that had simply never seen that session before.
In this illustrative failure, complaints cluster around deploys or scaling events that change routing or remove local state. That is a clue, not proof: load can also expose missing sessions, and a controlled two-server test can reproduce it without production traffic. Correlate the session’s creation and lookup servers and test a restart or destination change.
Stickiness is not the mistake by itself — it is a real technique and sometimes the right one, particularly as a short-term way to keep something working while state is moved somewhere better. What makes it the wrong answer in this shape is that it ended up doing load-bearing work nobody had actually decided to give it. The session data belongs somewhere both servers can reach, and stickiness should be free to switch off without anyone noticing, an optimisation rather than the thing quietly holding the system together.
When a request is retried against a different server, that server needs to handle the same thing twice, which is what that article is about. Of the roadmaps this site keeps, this page has the closest fit with the architecture path.
IF YOU REMEMBER ONE THING
The algorithm decides how requests get spread; it does not decide whether the servers receiving them are interchangeable. Round robin assumes requests cost about the same, least connections can be pulled toward a server that is failing fast, and a hash trades even distribution for a mapping that moves whenever the pool does. Whichever one you pick, the failure that actually reaches production is state sitting on one server that the balancer never knew was there.
Questions people also ask
5 QUESTIONSWhich load balancing algorithm should I use?
Round robin if requests cost roughly the same and the servers are interchangeable — it needs nothing else to work well. Least connections if request cost varies, when its connection count is a useful approximation to load; multiplexed or differently priced work can defeat that approximation. Reach for a hash only when the same client genuinely needs the same server; it is a stability choice, not a distribution one.
What are sticky sessions and should I use them?
Routing that sends the same client back to the same server, usually by a cookie or by hashing a client attribute such as an address. Use them when session state genuinely lives on one server and moving it is not practical yet; treat that as a stopgap, because the pool loses the freedom to treat servers as interchangeable.
What is the difference between layer 4 and layer 7 load balancing?
A layer 4 decision is made from connection-level information — addresses and ports — before anything about the request itself has been read. A layer 7 decision is made after the request is parsed, so it can route on a header, a path or a cookie. Hashing a header or cookie needs that layer-7 view; hashing the source IP address does not.
What happens to in-flight requests when a server is removed from the pool?
It depends entirely on how the removal is done. A pool that stops sending new requests to a server while letting its existing ones finish lets those requests complete normally. A pool that drops the server outright ends whatever that server was still doing, and the client sees the connection fail mid-request rather than a clean response.
Is a load balancer the same as a reverse proxy?
A reverse proxy is the broader shape: something that sits in front of servers and forwards requests on their behalf, which can also handle other things along the way. An HTTP reverse proxy can load-balance requests, but not every load balancer is an HTTP reverse proxy. Layer-4 forwarding and direct-server-return designs distribute traffic without parsing and proxying each application request.