Dead letter queues

The name suggests a single meaning — messages that failed — and that reading is what makes dead letter queues confusing to operate. A broker sends messages there for several unrelated reasons, most of which are not a consumer rejecting anything, and treating the queue as a uniform pile of failures leads directly to the wrong remedy applied in bulk.

Four ways in, only one of them a bug#

RabbitMQ enumerates its triggers, and the list is more varied than the name implies. A message is dead-lettered when a consumer negatively acknowledges it with the requeue parameter set to false; when it expires through a per-message time-to-live; when it is dropped because its queue exceeded a length limit; or when it has been returned to a quorum queue more times than the delivery limit permits.

Only the first of those is code declining to process something. The second is a message that was never picked up in time. The third is a queue that filled up, which is a capacity signal about the whole system rather than a statement about the message at all — the message was fine and simply arrived when there was no room. The fourth sits between the two: something kept failing and the broker stopped counting. Four different situations, one destination, and no way to tell them apart without looking at why each message arrived.

Two things about that list are worth holding on to. It is one broker’s list, not a definition: the triggers are RabbitMQ’s, other brokers dead-letter on their own conditions, and some systems have no broker-side concept of a dead letter queue at all, only a topic that consumer code writes to by convention. So the transferable claim is not the four entries — it is that your broker has a list, that it is longer than “the handler failed”, and that you should go and read it. And the second: telling them apart is not guesswork, because the broker writes down which one applied.

The queue is a diagnosis, not a fix#

RabbitMQ records it in an x-death header on the message itself, and the fields are exactly the ones the previous section says you need. A reason, whose value is one of rejected, expired, maxlen or delivery_limit — the four triggers, named. The queue it was dead-lettered from, the exchange it had been published to and its routing-keys. A count of how many times this has happened to this message from that queue for that reason, and the time it first did. Entries are ordered by recency, most recent first, so a message that has been round the houses carries its own history in order. Any triage worth automating reads that header first and branches on reason; anything that treats the queue as an undifferentiated pile is discarding the evidence the broker already gathered.

Because the reasons differ, the remedies differ, and the instinct to write a job that replays the whole queue on a schedule is the wrong shape for at least two of the four. A message dead-lettered by a length limit usually wants replaying, provided the handler treats a repeated delivery as routine — nothing was wrong with it. A message rejected because it is malformed will fail again on the next attempt and every attempt after that, and a timer that keeps feeding it back is a machine for burning consumer capacity on a message that can never succeed.

This is also why configuration is per queue rather than global. RabbitMQ attaches the dead letter exchange to each queue through a policy or a queue argument, and recommends policies over hardcoded arguments for a specific operational reason: arguments baked into the queue declaration require an application redeployment to change, while a policy can be updated on a running system. That distinction matters most in exactly the situation dead letter queues exist for — the middle of an incident, when the routing needs changing and a deploy is the slowest available tool.

Cycles, and what stops them#

Nothing in the topology prevents a dead letter route from leading, eventually, back to the queue it came from. RabbitMQ says so plainly — it is possible to form a cycle of message dead-lettering where the same message reaches the same queue twice — and then describes the guard: it detects a cycle and drops the message, provided there was no rejection in the entire cycle.

That condition is the part worth reading twice, because it draws the boundary of the protection. A loop driven by expiry, where messages keep timing out and being re-routed, is caught and broken. A loop where a consumer is actively rejecting at some point in the ring is not — the broker treats an explicit rejection as a real decision by real code, and declines to discard a message on the strength of it. So the automatic protection covers the accidental cycle and leaves the one your own retry logic built, which is the one more likely to exist. A rejection that always happens is not a transient failure, and the design question it raises is whether this work belonged behind a broker at all.