Long polling

You have a page that needs to update when something changes on the server, you already know that asking again and again on a short, fixed schedule is wasteful, and you have probably been told the honest answer is WebSockets and that anything simpler is obsolete. The real question underneath that advice is narrower: is the simple thing — one request, held open, answered late — good enough for what you are building? Most of the time it is, and this article is about the mechanism that makes it work, the comparison that tells you when it stops being enough, and the specific way it fails when it does.

What holding the request actually costs#

The mechanism starts from a limit in HTTP itself: on an ordinary request, a server has no way to hand the client fresh data the client did not ask for. The client has to ask, and the server has to answer that specific request — there is no separate channel it can use to speak first. Long polling does not get around that limit; it works inside it. The client makes an ordinary request. The server does not answer straight away — it holds the request open, unanswered, until there is something worth sending back or until it decides there is not going to be. From the client’s side nothing about the request looks unusual. It is a plain request that is simply slow.

Held requests are not held only by the server, though. Every hop between the client and wherever the answer eventually comes from now has one open request sitting on it — the client’s own connection pool, any proxy or gateway sitting in front of the server, and the server’s own handling of the request — and each of those hops copes with an open connection on its own terms, whether or not anyone building the page thought about them.

Three consequences fall out of that, and together they decide whether the approach works at all. Every intermediary on the path enforces its own idle timeout, closing a connection that has sat open with nothing sent for longer than it is willing to wait. If that timeout is shorter than how long the server means to hold the request, the intermediary closes the connection first, and the client sees exactly what a dropped connection looks like — nothing back, just gone — and has to treat that as a normal outcome of the pattern rather than as an error. Second, over HTTP/1.1 a browser also caps how many connections a page may have open to one host at the same time, and a held request occupies one of that small allowance for the whole time it stays open, alongside every other request the page is trying to make to the same host — over HTTP/2 that ceiling is largely beside the point, because the same traffic multiplexes as streams negotiated on one shared connection rather than counting against a per-host connection limit. Third, on the server, a request being held is a request the server is still serving: whatever unit of work a request costs there, this one is spending it for the length of the hold instead of for an instant, and what that unit actually is differs by runtime — a lightweight, suspendable task on one, a whole thread sitting idle on another.

The loop this produces has one shape regardless of what runs it, and the code implementing it has to treat its two outcomes as one and the same:

def poll_forever(url, on_message):
    while True:
        response = hold_open_request(url)
        # A message and a timed-out hold arrive as the same event here:
        # both mean the wait ended, and both fall through to the loop
        # re-requesting immediately below.
        if response.has_data:
            on_message(response.data)
        # no separate branch for "the hold just ran out" — there isn't one

A timeout and a message are the same event as far as the client is concerned — both mean the hold ended and it is time to ask again. Code that treats them as different events, succeeding quietly on one and raising an error on the other, is where the failure in the last section of this article starts.

When it is the right answer#

Set the four ways of getting an update from a server next to each other and the choice mostly makes itself.

The infrastructure column compresses more nuance than a few words can carry. Long polling needs the path to tolerate one request held open longer than a proxy’s usual idle allowance, and nothing more exotic than that — it is still plain HTTP throughout. Server-sent events need the same tolerance for an open connection, but only one of them per client instead of a held-then-reopened cycle. WebSockets need every hop on the path to support the protocol upgrade in the first place, which is a stronger requirement than either of the other two ever makes. Plain polling needs nothing unusual from anything, at the cost of never being cheap while nothing is happening.

Long polling wins, plainly, when updates are infrequent, when the traffic really is one-directional — server to client — and when adding a second protocol to the stack costs more than the inefficiency it would remove. It loses when updates come often enough that the overhead of a fresh request for every one of them dominates the exchange, or when the client needs to send data as freely as it receives it, at which point the traffic was never one-directional and no amount of holding a request open changes that.

The timeout nobody configured#

The failure starts with a hold set longer than a proxy’s idle timeout somewhere on the path — usually one nobody on the team configured or even knew was there. The proxy closes the connection once it has sat quiet past its own limit. The client sees a connection close with no response, and to a handler written to treat that as a failure, it is one: so it retries, immediately, because that is what a failure handler does.

Multiply that by every client on the estate and the shape of the problem appears. Each one held its request for roughly the same length of time before the same proxy cut it off, so each one now retries at roughly the same moment, and the cycle repeats in step across the whole fleet. The load this produces looks exactly like the polling storm long polling was adopted to get away from — except now it is driven by a timeout instead of by genuine traffic. A client that retries because a request was closed is asking the server to handle the same thing twice, which is a narrower version of the same problem that article is about.

The tell is what makes this diagnosable rather than mysterious: the request rate holds steady and has no relationship to how much is actually happening upstream. Traffic driven by real events is bursty — it rises and falls with whatever is being reported on. Traffic driven by a timeout is metronomic, because it is timed by the proxy’s clock rather than by anything the application produced. A constant, event-independent request rate is the signature of a timeout, not of load.

Only part of the fix lives on the server. The reliable fix is keeping the hold shorter than the shortest idle timeout anywhere on the path, and treating a closed hold as a routine outcome to retry calmly rather than as an error to retry on. The hard part is the first half of that sentence: knowing what is actually on the path. The proxy that ends up ending the request is frequently the one furthest from the team that owns the server, added for a completely unrelated reason, with a timeout nobody who wrote the long-polling handler ever saw.

Timeouts, retries, and the assumptions a request makes about the network it crosses are the recurring shape once more than one machine sits on the path — the distributed-systems path is where that shape gets traced in more places than this one request.