Change Data Capture

Reliably reacting to every change in your database by streaming its write-ahead log — why it beats dual writes and polling, and how it keeps search indexes, caches and other services in sync.

Change Data Capture (CDC) is a pattern for streaming every insert, update and delete in a database to other systems, in order, as it happens. It solves a problem you hit constantly once a system grows: a write lands in your primary database, and other things need to know — the search index must update, a cache must invalidate, an analytics pipeline must record it, another service must react. CDC turns your database's own change log into a reliable event stream so all of that stays in sync without bolting event-publishing onto every write path.

The problem it replaces

Two common approaches both have real flaws:

Dual writes — the app writes to the database and publishes an event. But these aren't atomic: if the DB write succeeds and the publish fails (a broker blip), the two systems silently drift apart. (The transactional outbox is one fix; CDC is another.)
Polling — periodically scan the table for rows changed since last time. It's laggy, misses deletes, and hammers the database.

CDC avoids both by reading the changes from the one place that already records every change, atomically: the database's write-ahead log.

How it works

Every durable database writes each change to a write-ahead log (WAL) before applying it — that's how it survives crashes. CDC tools (Debezium is the common one) tail that log and emit each change as an event, usually onto Kafka:

app ─► [ Postgres: WAL ] ─► [ Debezium reads the log ] ─► [ Kafka: change events ]
                                                              ├─► search indexer
                                                              ├─► cache invalidator
                                                              └─► analytics / other services

Because it reads the log the database already produces, CDC captures every change — including deletes — in commit order, without the application doing anything extra and without the drift of dual writes.

What it's for

Keeping a search index fresh. Stream product changes to Elasticsearch so search reflects the database without the app indexing on every write.
Cache invalidation. React to a row change by evicting the matching cache key — centrally, reliably.
Feeding data pipelines / warehouses. Replicate operational changes into analytics systems in near-real-time.
Decoupling services / the strangler pattern. Let other services react to another service's data changes without coupling to its write path — useful when breaking up a monolith.

Why it's clean

The source service doesn't change: it just writes to its database as before. Everything downstream subscribes to the change stream. Add a new consumer (a new index, a new service) without touching the producer — the same decoupling benefit as event-driven architecture, but the events are guaranteed to match the database exactly, because they are the database's changes.

What to watch

Eventual consistency. Downstream systems lag the source by the stream's delay — fine for search and caches, not for a read that must be immediately consistent with the write.
Operational weight. You're running a CDC connector plus Kafka — real infrastructure. For a single derived index at modest scale, an outbox or even a synchronous update may be simpler.
Schema changes ripple. A change to the source table shows up in the stream; consumers must tolerate it.

What to say in an interview

Reach for it when several systems must react to database changes reliably:

"The search index, cache and analytics all need to reflect changes to the products table. Rather than have the app dual-write (which can drift) or poll (which lags and misses deletes), I'd use change data capture — Debezium tailing Postgres's write-ahead log into Kafka — so every change is streamed in commit order. The search indexer and cache invalidator consume it independently. It's eventually consistent, which is fine for those, and I'd only take on the infrastructure once more than one system needs the stream."

That completes Advanced Topics — and the Core Concepts, Patterns, Key Technologies and Advanced Topics sections of the track. Head back to any section from the sidebar, or start at Networking Essentials.