Events, and what they promise

Adding an event feels like the cheap option. Nothing waits, nothing blocks, and the two services stop knowing about each other. What is easy to miss is that the event carries a set of promises to everyone downstream, and two of them are weaker than people assume — the fact cannot be withdrawn, and the delivery is not single. Both shape the consumer far more than the producer, which is why the cost shows up somewhere other than where the decision was made.

What an event promises, and what it cannot#

Martin Fowler’s definition of a domain event is one line and every word of it is load-bearing: it captures the memory of something interesting which affects the domain. A memory, of something that already happened. Not a request, not an intention, not a question — a record.

That tense is the whole design. A command asks a named service to do something and can be turned down; an event reports something finished and is addressed to nobody in particular. Nothing about it can be refused, because there is nothing left to decide. This is what makes adding a second consumer free: the producer does not learn about it, does not depend on it, and cannot fail because of it.

The same page is explicit about what follows, and it is the part people design around rather than with: the source data on an event is immutable — once you have created the event object, that source data cannot be changed. An event is not a row you correct later. If the order was confirmed and should not have been, the confirmation still happened; what you add is a second event that cancels it. Both facts stay in the log, in the order they became true, which is exactly what a consumer that already acted on the first one needs in order to act on the second.

So the promise is narrow and firm: this happened, and it will not stop having happened. Everything people expect on top of that — it happened once, you will hear about it once, you will hear about things in the order they occurred — is not part of it, and each of those has to be bought separately.

The broker promises delivery, not single delivery#

Message brokers typically provide at-least-once delivery, and the microservices.io pattern catalogue states both halves of what that buys. The guarantee is that the broker will deliver a message to a consumer even if errors occur — genuinely valuable, and the reason a queue survives a restart where a direct call would not. The stated side effect is that the consumer can be invoked repeatedly for the same message.

Read those two sentences together and the trade is visible. The broker cannot tell the difference between a consumer that never received a message and one that received it, processed it, and died before saying so. Both look identical from outside. Faced with that ambiguity it can either resend and risk a duplicate, or not resend and risk a loss — and at-least-once is the choice to resend. The duplicate is not a defect in the broker. It is the price of the guarantee, paid by the consumer.

Ordering is the second assumption worth checking, and it holds in a smaller box than most designs assume. Confluent’s guarantees course describes Kafka’s behaviour precisely: events are written to a particular partition in the order they were sent, and consumers read those events in the same order. Partition, not topic. Extend it with a key and the end-to-end version follows — events with a specific key always land in a specific partition in the order they are sent, and consumers always read them from that partition in that exact order.

Which makes ordering a design decision rather than a property you inherit. Related events stay in order because somebody chose a key that routes them to one partition — a customer id, an order id. Get the key wrong and two events about the same entity can be processed out of sequence while every individual guarantee is still being honoured.

Durability sits on the producer side of the same conversation, and it is likewise a setting rather than a given. Confluent describes three acknowledgement modes: acks=0, fire and forget, with no strong durability guarantee since the partition leader might never receive the data; acks=1, which confirms the write reached the leader replica but does not wait for followers; and acks=all, where the send is not acknowledged until the data is on the leader and all follower replicas in the in-sync set. Three different meanings of the word “sent”, and the one in force is whichever was configured, usually once, by whoever set the client up.

What the consumer must tolerate#

All of the above lands in one place, and it is not the service that raised the event. The requirement on the consumer, as the pattern catalogue states it, is that the outcome of processing the same message repeatedly must be the same as processing the message once.

Note what that sentence does and does not ask for. It does not say the handler must detect duplicates, or run only once, or be free of side effects. It constrains the outcome. A handler that sets a status to confirmed satisfies it without any machinery at all, because setting it twice leaves the same value. A handler that adds a line to a ledger does not, because running it twice leaves two lines and the second is money that does not exist.

For the handlers that need help, the catalogue’s mechanism is deliberately unclever: record the ids of processed messages in the database. A PROCESSED_MESSAGES table takes an insert of the message id inside the same transaction as the work; a repeat delivery fails that insert on the primary key constraint, and the handler rolls back and ignores the message. The duplicate is caught by a constraint the database was already enforcing, in the transaction that was already open — which is why it holds under concurrent redelivery, where a check-then-act written by hand would not.

This is the point on the roadmap where the obligation gets its own page. What a handler has to do to be safe when it is called again — the keys, the storage, the window over which a repeat is still recognised — is the subject of idempotency keys and safe retries, and it is the direct consequence of everything above. Events promise delivery; the consumer supplies the rest.

Two related decisions sit either side of it. Before adding the event at all, the question of whether the work belonged behind a broker is worth asking honestly — a queue moves work in time and does nothing else, and three common problems it gets handed are not that. And once messages start failing for good, they land in a dead letter queue, whose contents mix genuine bugs with expiry and capacity limits and want reading before they want replaying.

Where it goes wrong#

The failure worth recognising is not a duplicate that slips through. It is a system where every individual promise was kept and the outcome is still wrong, because the key was chosen without anyone noticing they were choosing an ordering guarantee.

The shape: a service publishes OrderPlaced and, seconds later, OrderCancelled. The producer sets the message key to the event’s own identifier, which is unique per event and therefore looks like an excellent key — it distributes perfectly across partitions, which is exactly what a key is usually chosen to do. Two events about the same order now land on two different partitions, and nothing in the system relates them any more. A consumer reading both partitions in parallel can process the cancellation first.

What happens next is the part that hides it. The cancellation handler looks for the order, does not find it, and — being defensive, as handlers written for at-least-once delivery are trained to be — treats the absence as a duplicate it has already dealt with and returns quietly. Then the placement arrives and creates the order. Nothing has failed. No exception is thrown, no message reaches a dead letter queue, no metric moves. There is simply a live order that somebody cancelled, and it stays live until a human is told about it by the customer.

Two things make this expensive out of proportion to the fix. The first is that it is invisible in exactly the environments where it would be caught: with one partition, or with low enough volume that the two events are processed in the order they were sent, the system behaves perfectly. It appears in production, under load, and disappears when anybody tries to reproduce it. The second is that the guilty line is a routing key set once, months earlier, in a producer configuration, while every symptom is in the consumer — so the investigation starts in the wrong service and reasonably concludes that the handler is buggy.

The remedy is the design decision named above rather than a repair: choose the key from the entity whose history has to stay in order — the order id, the customer id — so that everything about one entity lands on one partition. What is worth taking from the failure, though, is the general form. When the outcome is wrong and no guarantee was broken, the missing guarantee was one nobody bought.