Load balancers, concretely

Two servers sit behind something that spreads requests between them, and the question that actually matters is not which algorithm looks fastest on paper — it is whether the two servers are truly interchangeable. Get that part wrong and no algorithm saves you: users start getting logged out at random, nothing in the logs names a cause, and the balancer gets blamed for a decision it never made. This is what the three common ways of picking a server actually do, what each one assumes about the servers behind it, and the one assumption that breaks all three the same way.

Three ways to pick, and what each assumes#

Three of those four are choices; the last row is the baseline they are measured against, and a purely random pick is in the table for exactly that reason. Round robin is the simplest of the three and the easiest to reason about: it sends the first request to the first server, the next to the second, and starts over once it reaches the end. That only stays fair while requests cost roughly the same amount of work to serve. The moment one request is markedly more expensive than the others, round robin has no way to notice — it has already committed to the next server in line, regardless of how busy the current one turned out to be. The fast server ends up idling behind whichever one happened to draw the expensive request, not because it was unlucky but because rotation does not look at cost at all.

Least connections measures the exact thing round robin has no view of: it sends each new request to whichever server currently has the fewest connections open, which is a reasonable stand-in for how busy that server is. Its failure mode is the mirror image of round robin’s, and it is the sharper one to watch for. A server with a broken dependency that answers every request with an immediate error looks, by this measure, like the least busy server in the pool — its connections close fast because it isn’t doing any real work, and least connections keeps sending it more traffic precisely because it looks free. A server failing quickly can pull load toward itself instead of away from it.

Hashing a client attribute — an address, or something else the client presents consistently — buys a property neither of the other two has: the same client keeps landing on the same server. That is how sticky sessions are usually built, whether by hashing directly or by a cookie that records the choice the hash already made once. The trade is that even distribution is no longer the goal; consistency is, and consistency is only as good as the pool it was computed over. Add or remove a server and the hash’s output changes for a share of every client, not just for whoever was routed to the server that moved — because the mapping is a function of the whole pool, not of one client in isolation.

One distinction underlies why a hash can even be computed on a client attribute in the first place. Choosing a server from connection-level information — an address and a port, before anything about the request has been read — is a different amount of information from choosing after the request itself has been parsed. A hash on a header, a cookie or a path needs the second kind of decision; a hash on the connection’s own address does not. Which layer a balancer decides at is not a detail — it is the difference between having that information available to hash on at all and not.

What the health check actually proves#

A health check proves exactly what it asks and nothing more. An endpoint that reports healthy because the process is running and able to answer HTTP at all says nothing about whether that process can reach the database, the cache or whatever else it actually needs to do its job. A server with a dead dependency stays in the pool under a check like that, and answers every request it receives with an error — fast, because failing needs no real work, which under least connections is exactly the condition that draws more traffic toward it rather than away.

The fix is naming two different questions rather than asking one and hoping it covers both. A liveness check asks whether the process is alive at all — whether it should be restarted if it is stuck or has crashed. A readiness check asks something narrower and more useful to a balancer: can this specific instance serve a request right now. A pool that only ever asks the first question will happily route traffic to an instance that is alive, responding, and unable to do anything useful with what it receives — which is precisely the gap a check built only to catch a crashed process leaves open.

The honest complication runs the other way too. A readiness check that reaches all the way through to every dependency the server touches is more accurate about that one instance, but it also means a dependency shared by the whole pool — one thing every instance calls out to — can take every instance out of rotation at once if it wobbles. A shallow check misses a real problem; too deep a check turns one shared dependency’s bad moment into every server failing its check simultaneously, which is a total outage standing in for what was actually a partial one.

Logged out after every deploy#

The failure that actually reaches production starts with session state kept in one server’s own memory. It works, for a while — either every request from a given user happens to keep landing on the same instance by chance, or stickiness is switched on and nobody thought hard about what it was quietly holding together. Then a deploy rolls through, or the pool scales up, or a single instance restarts, and the mapping that stickiness depended on moves. Users are logged out at random. There is no error anywhere that names the cause, because nothing failed — the request that lost its session was answered correctly by a server that had simply never seen that session before.

The symptom worth writing down, because it is what makes this recognisable instead of mysterious, is that the complaint rate tracks deploys, not load. It does not rise with traffic and does not reproduce under stress in the usual sense — it spikes right after a release or a scaling event and is quiet in between. It is also close to invisible in staging, where there is often exactly one instance and the whole question of which server a request lands on never comes up.

Stickiness is not the mistake by itself — it is a real technique and sometimes the right one, particularly as a short-term way to keep something working while state is moved somewhere better. What makes it the wrong answer in this shape is that it ended up doing load-bearing work nobody had actually decided to give it. The session data belongs somewhere both servers can reach, and stickiness should be free to switch off without anyone noticing, an optimisation rather than the thing quietly holding the system together.

When a request is retried against a different server, that server needs to handle the same thing twice, which is what that article is about. Of the roadmaps this site keeps, this page has the closest fit with the architecture path.