What a Kafka topic actually is

The definition you have already read — a topic is a category or feed name that messages get published to — is true and decides nothing. It does not tell you what number to type into the partition count, it does not tell you what to use as a key, and it does not explain why a record you sent second was processed first. Those three questions have one root between them: the topic is not the thing Kafka makes promises about. The partition is, and the topic is only the name you gave a set of them. Everything you are about to configure follows from where that line falls.

A topic is a name over several logs#

Apache Kafka’s own introduction puts it plainly: topics are partitioned, meaning a topic is spread over a number of buckets, and a new event published to a topic is appended to one of the topic’s partitions. Read that in the other direction and the whole model falls out. The partition is the thing that exists — a file being appended to, in order, forever. The topic is a label over a group of them. There is no combined log underneath the label that the partitions are slices of; the partitions are all there is.

Each partition numbers its own records as they arrive, and that number is the offset. Kafka’s consumer documentation calls it a unique identifier of a record within that partition, and the consumer’s position in that partition — both halves of that sentence are scoped to one partition, and neither is scoped to the topic. So an offset is half an address. “Offset 400” names a record only once you have also said which partition, and every partition in the topic has an offset 400 as soon as it is long enough to have one, each pointing at something unrelated.

This is why a consumer’s progress is tracked per partition and never per topic. A consumer that reports it has read up to offset 400 has said nothing about the topic. It has said one thing about one partition. Its progress through the topic is not a number at all — it is a list of numbers, one per partition it is assigned, and those numbers move at completely different speeds because the partitions fill at completely different speeds.

The append-only part carries its own consequence. Reading a record does not consume it in the sense a queue would; the record stays where it was written and the consumer’s position moves instead. Kafka keeps records for as long as the retention policy configured on the topic says to, and discards them after that — which is why a consumer can be rewound and read the same records again, and why two consumer groups reading the same topic are not taking work from each other.

The key decides the order you get#

If a partition is where order lives, then the only question that matters when you produce a record is which partition it lands in — and that is what the key answers. Send with no key and the client’s default partitioning logic picks a lane without looking at the message at all. It does not spread every record evenly, and it does not keep choosing the same one; what it does not do is pay any attention to what is inside. Across the topic, the order you observe afterwards is not the order you produced, and no configuration will make it so.

Send with a key and, under that same default logic, the partition is chosen from a hash of the key. Every record carrying that key therefore lands in the same partition, behind the previous one, so all records for that key come back in the order they were sent. That is a smaller promise than “the topic is ordered”, and it is almost always the promise systems actually need: the account’s events in order, the device’s readings in order, this customer’s updates in order, with everyone else’s running independently alongside.

# The client's default partitioning logic picks the lane from the key:
# a hash of the key, over the topic's partition count at send time.

producer.send("orders", key=b"acct-a", value=b'{"op": "create"}')
# -> some lane. Which one is not worth knowing.

producer.send("orders", key=b"acct-a", value=b'{"op": "update"}')
# -> the SAME lane, appended after the create. This pair is ordered.

producer.send("orders", key=b"acct-z", value=b'{"op": "create"}')
# -> a lane chosen independently. It may even be the same lane by
#    coincidence, and that would still guarantee nothing: no ordering
#    is promised between acct-z's records and acct-a's.

producer.send("orders", value=b'{"op": "ping"}')
# -> no key, so the client picks a lane without reading the message.
#    Nothing connects this record's position to the ones above it.

No comment above names a partition number, because none of them could. The lane a key lands in depends on the hash and on how many partitions the topic has when the record is sent — the only durable fact is same key, same lane.

Two consequences follow from that, and both bite in production. The first is that the mapping from key to partition is computed against the partition count, so the count is part of the mapping rather than a setting beside it. Kafka’s operations documentation is direct about this: adding partitions does not change the partitioning of existing data, and if data is partitioned by a hash of the key modulo the number of partitions, that partitioning is potentially shuffled by adding partitions — and Kafka will not attempt to redistribute anything. Old records stay where they are. New records with the same key can land somewhere else.

The second is about how many consumers can usefully read. Within one consumer group, partitions are balanced across the members so that each partition is assigned to exactly one consumer in that group. That is a statement about a single group — other groups read the same partitions at the same time, entirely independently — but inside the group it sets a ceiling. Add more members than there are partitions and the extra members sit idle, because there is nothing left to assign them. The partition count is not a performance dial you can turn; it is the number of workers a group is allowed to have.

When the partition count changes#

Consumer lag on one topic is growing. The group is already running one member per partition, so the obvious move is the one that gets made: add partitions, add consumers, watch the lag come down. It does come down. The change looks like a success for as long as it takes for the first ordering-sensitive key to be affected, which may be minutes and may be a fortnight.

None of this adds up to “never add partitions”. Sometimes you have to, and the metric it improves is real: consumer parallelism inside a group genuinely is capped by the partition count, so widening the topic genuinely does let more members work. That is exactly why it gets misread as a capacity dial. It is a data-migration event wearing a dial’s clothing, and it wants the treatment a migration gets — drain the affected keys, or stop producing while the partitions are added, or accept a stated window in which per-key ordering does not hold and make sure the consumers can survive it.

Surviving it usually means the consumers stop depending on order for correctness. A handler that can apply the same record twice, or apply an update whose create has not arrived yet, without corrupting anything, is one that a partition change cannot silently break. That is the same property an order endpoint needs when a client retries: a consumer that sees the same message twice has the problem idempotency keys and safe retries is about. It is also the honest boundary of what a broker can do for you. Kafka can promise things about what its own log contains and what order a partition returns; it cannot make a row appear in your database or an order get placed exactly once, because neither of those is a thing Kafka does. That part is the consumer’s to guarantee, every time.

Ordering, retries and the duplicates they produce are the recurring shape once more than one machine is involved. The distributed-systems path gathers them in the order that makes each one explain the next.