Change data capture
See why reading the database's own log beats polling for changes, what a replication slot guarantees, and the way that same slot can take the server down.
Log-based CDC reads a database change stream instead of polling rows. PostgreSQL can decode its write-ahead log; MySQL capture commonly reads the binary log, separate from InnoDB's crash-recovery redo log. Ordering and delivery guarantees depend on the database and connector.
The till roll is already printing. Walking the shelves each evening to guess what moved is the slow way round; reading the roll gives you every sale in order, including the things a shelf count can never show you, like an item rung up and then refunded.
Every system eventually needs a second copy of its data somewhere else — a search index, a warehouse, a cache, another service. Polling tables for recent changes can miss deletes and intermediate updates. Log-based capture reads a suitable change stream instead; which log exists and what it guarantees depend on the database and connector.
Reading the log instead of the table#
Log-based change data capture reads a stream of committed database changes rather than inferring them from the current contents of a table. PostgreSQL’s logical decoding reads its write-ahead log. MySQL CDC commonly reads the binary log used for replication, while InnoDB crash recovery uses a separate redo log. The log and its retention, ordering and replay contract belong to the specific database and connector, not to relational databases as a class.
In PostgreSQL, WAL is also used for crash recovery and describes storage-level changes rather than a ready-made row-event feed. PostgreSQL’s logical decoding translates persistent WAL changes into an application-specific form such as tuples or SQL statements. MySQL uses a different split: its binary log supplies replication and many CDC connectors, while InnoDB uses a separate redo log for crash recovery.
Compare that with polling a table for recently updated rows. Timestamp polling cannot see a deleted row or intermediate states between polls, and it depends on every writer maintaining the column. A suitable change stream can expose those events, but consuming it correctly still requires attention to retention, connector restarts, duplicate delivery and transaction ordering. Do not infer exactly-once consumer processing solely from a database’s durability mechanism.
The slot is the contract#
A consumer that reads a log needs to know where it got to, and that bookkeeping cannot live only in the consumer — if it did, a consumer that fell over would have no way to learn what it missed while the database recycled the segments. PostgreSQL puts the bookmark on the server side, in a replication slot, described as a stream of changes that can be replayed to a client in the order they were made on the origin server.
Two guarantees come with it and both matter for what you can build. Changes arrive in the order they were made rather than the order they happened to be written, so a consumer never has to reason about a change belonging to a transaction that later rolled back. And a logical slot emits each change just once in normal operation, which is what makes it reasonable to treat the stream as a sequence of facts rather than as hints to be re-checked against the source.
Slots outlive their consumers#
The server-side bookmark is the strength and the hazard, and PostgreSQL’s documentation is unusually blunt about the second. Slots persist across crashes and know nothing about the state of their consumers. They prevent removal of required resources even when no connection is using them — so write-ahead log segments cannot be recycled and vacuum cannot clean up the rows the slot still needs. The warning ends where it has to: in extreme cases this could cause the database to shut down to prevent transaction ID wraparound, so a slot that is no longer required should be dropped.
Read that as an operational rule rather than a footnote. A capture pipeline switched off for an afternoon, a consumer pointed at a new environment, a proof of concept nobody cleaned up — each leaves a slot behind, and each of those slots quietly instructs the database to keep everything the slot might still want. The failure surfaces as a disk filling on the primary, which is a long way from where the change was made and looks nothing like a consumer problem.
The subtler issue is what the stream means to whoever receives it. A row-level change feed describes what the storage did, not what the business decided — an update to three columns is three column values, not the intention behind them. Consumers written against that shape end up reconstructing intent from diffs, and they break whenever the schema changes underneath. This is the point at which the honest question is whether these two systems should be exchanging events they both understand instead of one of them reading the other’s storage.
IF YOU REMEMBER ONE THING
Log-based CDC can capture changes that table polling misses, but its source and delivery guarantees are database-specific. In PostgreSQL, a replication slot retains resources for a consumer even when that consumer is disconnected; monitor or remove unused slots.
Questions people also ask
4 QUESTIONSWhy not just poll for rows with an updated_at column?
Timestamp polling can miss deletes and intermediate changes between polls, and every writer must maintain the timestamp. A suitable database change log can record those events, including deletes, but the exact log and delivery guarantees depend on the database and capture connector. PostgreSQL logical decoding reads WAL; MySQL CDC typically reads the binary log.
What does logical decoding actually produce?
PostgreSQL describes it as decoding the contents of the write-ahead log, which describe changes at a storage level, into an application-specific form such as a stream of tuples or SQL statements. The log itself is written in terms of pages and byte offsets; decoding turns that into changes to rows that a consumer can understand without knowing the internals.
In what order do changes arrive?
PostgreSQL's replication slot represents a stream of changes that can be replayed in the order they were made on the origin server, and a logical slot emits each change just once in normal operation. Commit order rather than the order writes happened to hit the log is what makes the stream usable — a consumer never sees a change belonging to a transaction that later rolled back.
Can a capture pipeline take down the database it reads from?
Yes, and the documentation warns about it directly. Slots persist across crashes and know nothing about the state of their consumers, so they prevent removal of the resources they still require even when nothing is connected. That retains write-ahead log segments and blocks vacuum, and PostgreSQL notes that in extreme cases this could shut the database down to prevent transaction ID wraparound.