What Is CDC? Change Data Capture Explained

Ask most engineers how two systems stay in sync and they will describe a nightly batch job that copies a whole table from one place to another. It works, until the tables get large, until the business wants yesterday's data to be this minute's data, and until the batch window stops being long enough to finish before the next one starts. Change data capture, almost always shortened to CDC, is the answer that the data world converged on. Instead of copying everything on a schedule, it captures each individual insert, update and delete as it happens and streams just those changes to wherever they are needed. This guide explains CDC clearly, from the problem it solves to the mechanics of how it reads a database transaction log, and it sits inside a broader story about how enterprise systems talk to each other. If you want that wider map first, start with the pillar on enterprise system integration explained, then come back here for the data-movement layer of it.

The idea in one line: CDC turns a database from something you have to poll into something that tells you what changed, the moment it changes. Everything else in this article is a consequence of that shift. For where CDC fits among the other integration patterns, the integration pillar is the companion piece.

1. What CDC is

Change data capture is a technique for identifying and delivering the changes made to a source database so that other systems can react to those changes. The unit of work is not a table or a file, it is a single row-level event: this row was inserted, this row was updated from these old values to these new values, this row was deleted. CDC observes those events at the source and makes them available as a continuous stream to one or more downstream targets.

The important word is changes. A traditional replication or export job asks a broad question, "give me the current state of this data", and hauls back everything whether it moved or not. CDC asks a much narrower and much cheaper question, "what has changed since I last looked", and moves only the answer. On a table with ten million rows where two thousand changed today, the difference between copying ten million rows and shipping two thousand change events is the difference between a job that strains the system and a stream that barely registers.

CDC is also, at its best, near real time. Because the changes are captured as they are committed rather than gathered on a timer, a well-built CDC pipeline can deliver a change to a downstream system within seconds of it happening at the source. That latency profile is what lets CDC power things a nightly batch never could: live dashboards, cache invalidation, search indexes that stay current, and event-driven microservices that react to data changing rather than asking repeatedly whether it has.

2. The problem it solves

To appreciate CDC you have to feel the pain of the alternatives, because CDC exists to remove specific, concrete costs that older approaches impose. There are broadly three older ways to move data between systems, and each has a failure mode that CDC addresses directly.

Full reload is the simplest and the most brutal. You truncate the target and copy the entire source table into it on a schedule. It is easy to reason about and it self-heals any drift, but it does not scale. As the table grows, the copy takes longer, consumes more network and storage bandwidth, and hammers the source database with a heavy read every cycle. Eventually the reload takes longer than the interval you wanted to run it at, and you are stuck.

Polling for changes is the next step up. You add a column such as last_modified and repeatedly query for rows changed since your last checkpoint. This moves far less data than a full reload, but it puts the source under a constant drip of queries, it struggles to detect deletes because a deleted row is simply gone and never shows up in a query, and its freshness is limited by how often you are willing to poll. Poll every five minutes and your data is up to five minutes stale; poll every five seconds and you are back to loading the source.

Application-level dual writes, where the application writes to the database and then also pushes the same change to a second system, look appealing until the second write fails while the first succeeds. Now the two systems disagree and there is no clean way to reconcile them. This is one of the most common sources of silent data corruption in distributed systems.

CDC solves the problem those three share: efficient, complete, low-latency incremental data movement. It moves only what changed, so it is efficient. It captures deletes as first-class events, so it is complete. It reacts to commits rather than a timer, so it is fresh. And when it reads from the transaction log, it imposes almost no additional query load on the source, so it is gentle on the system it is watching.

3. How CDC works

The most powerful form of CDC does something clever: it does not read your tables at all. Instead it reads the database's own transaction log. Every serious relational database keeps a durable, ordered record of every change it commits, because that log is how the database recovers after a crash and how it replicates to standbys. In SQL Server it is the transaction log, in PostgreSQL it is the write-ahead log, in MySQL it is the binary log, in Oracle it is the redo log. The names differ, the principle is identical: an append-only sequence of every committed change, in commit order.

Log-based CDC attaches to that log as if it were another replica. It reads the stream of committed changes, decodes each entry into a structured event describing the table, the operation, and the before and after row values, and forwards those events to the targets. Because it is reading a log the database is already writing for its own durability, it adds almost no load to the source and it never misses a change or interferes with the transactions producing them. The diagram below shows the shape of it: one source, its log, a capture process, and several very different targets fed from the same stream.

The mental shift worth internalising is that CDC decouples the source from the targets completely. The source database does its normal work and writes its normal log. The capture process, running separately, turns that log into a stream. And any number of targets can subscribe to that stream without the source knowing or caring how many there are or what they do with the events. Add a new target next year and the source is untouched.

4. CDC methods

Log-based CDC is the gold standard, but it is not the only way to capture changes, and the older methods still show up in real systems, often because a particular database or a particular permission constraint rules the log-based approach out. It is worth understanding the three main families, how each one works, what it costs the source, and where each one falls down.

The pattern in that table is consistent. Log-based CDC pushes the cost onto a mechanism the database already runs for its own sake, so it is the lightest on the source and the most complete, at the price of needing deeper access to the database internals. Trigger-based CDC buys completeness and portability by paying a write-time tax that grows with your transaction volume. Query-based CDC is the easiest to bolt on with nothing but SQL, and it is the leakiest: it silently misses deletes and it is only as fresh as its polling loop. If you can use log-based CDC, you almost always should, and the rest of the industry has voted with its tooling.

5. CDC, ETL and event streaming

CDC is often confused with the things it sits next to, so it helps to place it precisely. CDC is not ETL, and it is not event streaming, but it plugs into both, and the confusion usually comes from the fact that a real pipeline uses all three together.

ETL and ELT describe the overall job of getting data from operational systems into an analytics store and shaping it for reporting. Traditionally that job was batch: extract a chunk, transform it, load it, on a schedule. CDC changes the extract step from a periodic bulk pull into a continuous change stream, which is what lets a warehouse move from nightly freshness to near real time. In other words, CDC is frequently the modern extract mechanism inside an ELT pipeline rather than a competitor to it. If the batch-versus-streaming and transform-order questions are what you are weighing, the ETL vs ELT comparison is the piece to read alongside this one.

Event streaming is the transport and the pattern. Once CDC has turned database changes into events, those events usually flow through a streaming platform such as a log-structured message broker, where they can be buffered, replayed and consumed by many independent services. CDC is a producer of events; event streaming is how those events are carried and consumed at scale. The two are so complementary that CDC is one of the most common ways to get data into a streaming platform in the first place. For the transport side of the story, see what is event streaming.

A clean way to hold the three in your head: ETL and ELT are the why (get data somewhere useful and shape it), CDC is the what changed (capture the source deltas efficiently), and event streaming is the how it travels (carry those changes reliably to many consumers). A modern data platform typically uses CDC to feed a stream that feeds an ELT process, and each layer does one job well.

6. Real use cases

CDC is not an academic pattern; it is load-bearing infrastructure under several everyday capabilities. The clearest way to understand its value is to see the concrete jobs it does.

• Real-time analytics: streaming committed changes into a data warehouse or lakehouse keeps dashboards and reports current to within seconds instead of a day behind. Sales, operations and finance stop looking at yesterday and start looking at now, without the nightly batch load ever running.
• Database replication and migration: CDC keeps a second database continuously in sync with a first, which is how zero-downtime migrations work. You seed the new system, let CDC stream every ongoing change into it until the two are identical, then cut over with no data loss and almost no outage.
• Cache invalidation: caches are famously hard because the cache does not know when the underlying data changed. CDC solves it directly. When a row changes, the change event tells the cache exactly which key to invalidate or refresh, so the cache stays correct without guessing or blanket expiry.
• Microservices and event-driven architecture: in a system where each service owns its own database, CDC lets one service publish its data changes as events that other services react to, without those services reaching into its database. It is the safe way to share state across service boundaries, and it avoids the dual-write trap entirely by deriving the events from the committed database changes themselves.

The thread running through all four is the same: something downstream needs to know that data changed, and CDC is the reliable, low-latency way to tell it. Whether the downstream thing is a report, a replica, a cache or a service, the capture mechanism is identical, which is exactly why CDC is worth learning once and reusing everywhere. In the enterprise application world, the same instinct shows up when you decide how to keep a business system in sync; see how the trade-offs play out in Business Central APIs and integrations.

An honest caution: CDC delivers changes, it does not deliver correctness for free. Downstream consumers still have to handle events arriving out of the order they expect during recovery, handle the same event being delivered more than once, and handle the initial bulk load of existing data before the stream begins. CDC gives you a faithful feed of what changed; building consumers that stay consistent under failure is still real engineering, and skipping that work is how teams get burned.

7. Challenges

CDC earns its reputation, but it is not free of sharp edges, and the teams who succeed with it are the ones who plan for the hard parts up front rather than discovering them in production. Three challenges come up on almost every project.

Schema changes are the most common source of surprise. The moment someone adds a column, renames a field or changes a type on the source, the change events start carrying a shape the downstream consumers were not built for. A mature CDC pipeline treats schema evolution as a first-class concern: it propagates schema changes as their own events, versions the event format, and gives consumers a defined way to adapt. Ignore this and a routine migration on the source silently breaks every downstream target.

Ordering and delivery guarantees matter because the whole value of CDC is that the changes arrive in a form the target can trust. Within a single table or key, changes must be applied in commit order, or an update can land before the insert it depends on. Across the failures that inevitably happen, most systems provide at-least-once delivery, which means the same change can arrive twice, so consumers need to be idempotent, able to process the same event twice without doing damage. Getting exactly-once semantics is possible but costly, and pretending you have it when you do not is a classic way to corrupt a target.

Initial load, sometimes called the snapshot, is the unglamorous part everyone underestimates. CDC captures changes from the moment it starts, but the target usually needs the existing data too. So a real deployment begins with a bulk snapshot of the current state and then switches to the change stream, and the handoff between the two has to be seamless: no rows missed in the gap, no rows double-counted in the overlap. Doing that snapshot on a large, busy table without locking it or falling behind the live stream is genuinely one of the trickier engineering problems in the whole pattern.

8. When to use CDC

CDC is powerful, which means it is also possible to reach for it when something simpler would do. The honest guidance is to match the tool to the requirement rather than adopting CDC because it is fashionable.

Reach for CDC when freshness genuinely matters and the data volume makes full reloads impractical: when a warehouse needs to be minutes fresh instead of a day, when a cache or search index must reflect changes quickly, when you are replicating or migrating a live database with minimal downtime, or when microservices need to react to each other's data changes without sharing databases. In all of these the combination of low latency and low source overhead is exactly what CDC provides and nothing simpler does as well.

Do not reach for CDC when a nightly batch is genuinely good enough, when the tables are small enough that a full reload is trivial, or when you do not have the operational maturity to run a streaming pipeline and handle the ordering, idempotency and snapshot concerns above. A well-run batch job that meets the actual freshness requirement is not a failure to modernise; it is the right tool for a requirement that does not need real time. CDC is worth its complexity precisely when the requirement outgrows the batch, and adopting it before that point buys you operational burden without a matching payoff.

9. References

The concepts in this article are drawn from well-established, publicly documented data-engineering practice rather than any single proprietary source. A few pointers worth naming for further reading:

• Log-based CDC concepts are grounded in how relational databases maintain a durable, ordered transaction log for recovery and replication: the write-ahead log in PostgreSQL, the binary log in MySQL, the transaction log in SQL Server and the redo log in Oracle. The vendor documentation for each of these logs is the primary reference for how change data is physically recorded.
• Debezium is a widely used open-source CDC platform that reads database transaction logs and emits change events, and its documentation is a practical, vendor-neutral reference for log-based CDC patterns, connector behaviour, snapshotting and schema-change handling.
• Streaming platform documentation for log-structured message brokers describes how the change events CDC produces are transported, retained and consumed, and covers the delivery-guarantee and ordering semantics discussed in the challenges section.

Wherever this article states a specific behaviour, such as query-based CDC being unable to detect deletes or log-based CDC imposing minimal source overhead, that behaviour follows directly from how the underlying mechanism works and is reflected consistently across the references above.

Final thoughts

Change data capture is one of those techniques that, once you understand it, you start seeing everywhere: behind the dashboard that is always current, behind the migration that happened with no downtime, behind the cache that never serves stale data, behind the microservices that stay in step without sharing a database. The core idea is small and durable. Stop copying everything on a timer and instead capture each committed change at the source, ideally by reading the log the database already keeps, and stream just those changes to whoever needs them.

What separates a CDC project that succeeds from one that frustrates is not the capture mechanism, which the tooling now handles well, but the discipline around it: choosing log-based capture where you can, designing consumers that tolerate re-delivery and out-of-order recovery, planning the schema-evolution story before the first migration, and getting the initial snapshot right. Treat CDC as the data-movement layer of a wider integration strategy rather than a magic sync button, and it becomes one of the most reliable tools in the kit. If you want the map that shows where this layer fits among APIs, messaging and the rest, the enterprise system integration pillar is the place to see the whole picture.

Related reading: Enterprise system integration explained, Change data capture in SQL Server, What is event streaming, ETL vs ELT, Business Central APIs and integrations.