Events, and what they promise
Find out what an event actually guarantees once it leaves the service that raised it, and which of those promises the consumer has to absorb itself.
An event records something reported to have happened. Its meaning does not itself guarantee delivery, order or one execution of a handler. With at-least-once delivery, design for redelivery and check the broker, producer and consumer settings that define the guarantee's boundary.
A postcard says a parcel was sent. Its contents and the postal service's delivery policy are different promises. A duplicate card must not make you charge twice, and a mistaken card needs an explicit correction.
An event can separate when a producer reports an occurrence from when a consumer acts on it. The producer may still wait for a durable publish acknowledgement, and both sides still share a data contract. Separate three questions: what the event means, how the transport delivers it, and what repeated processing does to the business state.
What an event promises, and what it cannot#
Martin Fowler’s definition of a domain event is one line and every word of it is load-bearing: it captures the memory of something interesting which affects the domain. A memory, of something that already happened. Not a request, not an intention, not a question — a record.
A command asks for work; an event reports an occurrence. Declining to process an event does not undo the occurrence, but a consumer can reject malformed or unsupported data. Adding a subscriber can leave producer code unchanged while still requiring schema compatibility, permissions, retention and capacity. That is useful decoupling, with real obligations.
Fowler’s model keeps source data immutable while allowing separate processing data. Corrections therefore need an explicit later record, linked to what it corrects. That does not ensure arrival or processing order: a consumer must tolerate a correction arriving before the original, according to its versioning and business rules. An immutable report can also have been false when written.
An immutable record can still contain a mistake. Keep the original report and make corrections explicit, with identifiers linking them where necessary. The event’s meaning supplies no transport guarantee: delivery, redelivery and ordering belong to separate contracts.
Two separate design choices sit outside that transport contract. Event sourcing uses a retained business-event history to rebuild state; ordinary event publication does not require it. Before delivery starts, a transactional outbox saves the pending message with its local state change, so a crash between commit and publish leaves recoverable work.
The broker promises delivery, not single delivery#
Under an at-least-once contract, an accepted message can be delivered again until acknowledged. Check the conditions: durable acceptance, retained data, recoverable failures and acknowledgement rules. An event that never reaches the broker, expires under retention, or is discarded by a configured limit is not rescued by the word “event.”
Lose the acknowledgement
An accepted event E42 adds one ledger line. The database commit succeeds, but the broker never receives the acknowledgement. Predict what a redelivery will do.
Consumer policy: Append a ledger line on every delivery, including a redelivery of E42.
- First delivery commits a ledger line, then its acknowledgement is lost.
- The broker redelivers E42. Blind append produces two lines; a transactional processed-id check keeps one.
The broker sees no acknowledgement in either case. It can redeliver even though the first commit succeeded. Enable JavaScript to step through both policies.
One accepted message, one subscriber and one database. The processed id and ledger change commit together, and the id remains stored. This does not prove delivery under every outage or make an external payment or email atomic.
Look at that from the broker’s side and the trade is visible. The broker cannot tell the difference between a consumer that never received a message and one that received it, processed it, and died before saying so. Both look identical from outside. Faced with that ambiguity it can either resend and risk a duplicate, or not resend and risk a loss — and at-least-once is the choice to resend. The duplicate is not a defect in the broker. It is the price of the guarantee, paid by the consumer.
A Kafka topic contains separate partition logs; records have a total log order inside each partition. An entity key can route related records together under a stable partitioning scheme; it does not establish occurrence-time order across producers or completion order when handlers run concurrently. Kafka’s producer contract also warns that retries with idempotence disabled and multiple in-flight requests can reorder batches within a partition. Partition-count changes can alter key mapping.
Which makes ordering a design decision rather than a property you inherit. Related events can stay in order only if somebody chose a key that routes them to one partition — a customer id, an order id — and the producer and consumer settings preserve that order. Get the key wrong and two events about the same entity can be processed out of sequence while every individual guarantee is still being honoured.
Durability sits on the producer side of the same conversation, and it is likewise a setting rather than a given. Confluent describes three acknowledgement modes: acks=0, fire and forget, with no strong durability guarantee since the partition leader might never receive the data; acks=1, which confirms the write reached the leader replica but does not wait for followers; and acks=all, where the send is not acknowledged until the data is on the leader and all follower replicas in the in-sync set. Three different meanings of the word “sent”, and the one in force is whichever was configured, usually once, by whoever set the client up.
What the consumer must tolerate#
All of the above lands in one place, and it is not the service that raised the event. The requirement on the consumer, as microservices.io’s idempotent-consumer pattern states it, is that the outcome of processing the same message repeatedly must be the same as processing the message once.
Note what that sentence does and does not ask for. It does not say the handler must detect duplicates, or run only once, or be free of side effects. It constrains the outcome. A handler that sets a status to confirmed satisfies it without any machinery at all, because setting it twice leaves the same value. A handler that adds a line to a ledger does not, because running it twice leaves two lines and the second is money that does not exist.
The idempotent-consumer pattern records a unique pair of subscriber id and message id in the same database transaction as the business change. A duplicate fails that insert, so the transaction adds no second effect. Commit before acknowledging, and retain the id for the required redelivery window. This transaction does not include a remote payment or email: those need their own idempotency contract or coordinated delivery design.
The keys, durable storage and window over which a repeat is still recognised are the subject of idempotency keys and safe retries. Delivery and effect guarantees must be designed together.
Two related decisions sit either side of it. Before adding the event at all, the question of whether the work belonged behind a broker is worth asking honestly — a queue moves work in time and does nothing else, and three common problems it gets handed are not that. And once messages start failing for good, they may be routed to a dead letter queue, if one is configured, whose contents mix genuine bugs with expiry and capacity limits and want reading before they want replaying.
IF YOU REMEMBER ONE THING
An event reports an occurrence. Broker settings define a delivery boundary; routing and producer settings shape order; consumer state and external-system contracts shape repeated effects. Review all three instead of inferring them from the word “event.”
Where it goes wrong#
The failure worth recognising is not a duplicate that slips through. It is a system where every individual promise was kept and the outcome is still wrong, because the key was chosen without anyone noticing they were choosing an ordering guarantee.
The shape: a service publishes OrderPlaced and, seconds later, OrderCancelled. The producer uses each event’s unique identifier as the message key. Kafka’s default partitioner chooses a partition from the key’s hash. Different keys, even with different hash values, can map to the same partition. Unique event identifiers therefore do not guarantee that an order’s events stay together. For this hypothetical example, assume the two keys map to different partitions. A consumer reading both partitions in parallel can process the cancellation first.
What happens next is the part that hides it. The cancellation handler looks for the order, does not find it, and — being defensive, as handlers written for at-least-once delivery are trained to be — treats the absence as a duplicate it has already dealt with and returns quietly. Then the placement arrives and creates the order. Nothing has failed. No exception is thrown, no message reaches a dead letter queue, no metric moves. There is simply a live order that somebody cancelled, and it stays live until a human is told about it by the customer.
This is a hypothetical failure, and a controlled test can reproduce it by delivering the cancellation first. Low traffic does not make it impossible. Review both routing and handler rules: absence is not evidence that a cancellation was already processed.
Choose a stable entity key where partition order is needed, configure producer retries appropriately and preserve that order through processing. Where late or out-of-order records remain possible, define an explicit version or state-transition rule instead of treating every absent entity as a duplicate. A routing key is one part of the design, not a complete end-to-end guarantee.
Questions people also ask
5 QUESTIONSWhat is the difference between an event and a command?
A command asks for an action; an event reports an occurrence. Consumers can reject an invalid event or decline to act without undoing the reported occurrence. A new subscriber may leave producer code unchanged, but still needs a compatible schema, access, capacity and sometimes historical data.
Can I take an event back if it was raised in error?
No, and the design is better for it. The event recorded that something happened, and a later discovery that it should not have happened does not unmake it. The correction is a second event that says what changed — a cancellation, a reversal, an amendment — which leaves both facts in the log where a consumer that already acted on the first can see the second.
Does a broker keep my events in order?
Kafka orders records within each partition, not across partitions. That is log order, not necessarily business occurrence time or completion order of concurrent handlers. Stable entity keys help routing, but producer retry settings, changes to partition mapping and consumer processing also matter. Other brokers have different contracts.
If delivery is at-least-once, why not just use exactly-once?
Because the guarantee usually stops at the boundary of the messaging system, and the side effect you care about is normally outside it — a charge at a payment provider, a row in another team's database, an email. A handler that produces the same outcome when it runs twice gives you the result people mean by exactly-once, and it keeps working when the broker's own guarantee does not reach far enough.
Where do events that cannot be processed end up?
If configured, a broker may route it to a dead-letter queue or another error path. Reasons and support vary by broker; some include expiry or queue limits as well as rejected processing. Reading why each message arrived is what decides the remedy, because replaying a message that failed on its own contents just sends it straight back.