Change Data Capture (CDC)

Change Data Capture (CDC): what is it and why it's important?

In most companies, the journey to analytics starts with a crime.

Not corporate espionage or anything that'd make headlines — just the quiet abuse of your transactional database to run nightly ETL jobs that sound innocent until you realize you're hammering the same tables your app depends on to stay alive.

The Crime Story

Imagine we're building a simple web application. Could be a CRM, ERP, a banking backend, billing system, or even POS terminals — pick your project.

At first, it's clean. Compact. A tidy little monolith backed by a relational database — PostgreSQL (maybe MySQL if you're feeling reckless). It does what it's supposed to: handles inserts, updates, deletes. Users log in, click around, the system responds. Nothing fancy.

But sooner or later, someone outside the engineering team starts asking questions. Questions like, "How many active users do we have?" or "What's our average time to close a ticket?" or the classic: "Why doesn't the report match what I see in the app?"

And just like that, the analyst shows up.

Suddenly, your transactional database isn't just for transactions. It's now the source of truth for everything. Everyone wants a piece — BI tools, ML pipelines, reporting systems, etc. Business wants data insights (whatever that means today). Ops wants real-time dashboards. Management wants today's metrics yesterday.

Data eventually becomes mission-critical. So the company starts running regular reports — because decisions have to come from somewhere, and that somewhere is your already-struggling production database.

Thing is, these reports? They're heavy. They run complex queries. They pull wide joins. They scan gigabytes/terabates/petabytes of junk data. These queries run nightly — or worse, hourly — and they drag your transactional system down with them.

They don't just slow things down. They burn CPU, eat memory, and chew through disk I/O. And that's before we even get into the network traffic from exporting all that data. Meanwhile, your business users are flying blind because the data they're waiting for only refreshes once a day. Which means you're always making decisions based on what happened yesterday, not today.

Sure, if you've got a quiet window — say 2 AM to 5 AM — and your nightly dump fits in that slot, you might get away with it. But what if your app has no off-hours? What if your data grows, and your window shrinks?

What happens when your extract job starts overlapping with business hours?

That's when the wheels come off.

Change Data Capture

Change Data Capture (CDC) is exactly what it sounds like: capturing only the changes. Not full tables. Not yesterday's copy of everything. Just the deltas — inserts, updates, deletes. The meaningful stuff.

Here's the idea: CDC watches your source system, sees what changed, and replicates just those changes to a target — usually a data warehouse or some other analytics store. From there, you do your reporting, analysis, forecasting, machine learning, late-night executive dashboards — you name it. And you do all of it without touching the operational database.

So your users keep working, your application keeps responding, and your analysts get fresh numbers without breaking anything. Everybody wins.

At its core, CDC provides a historical stream of changes to your tables. It doesn't just say, "Hey, something changed". It says what changed — which row, which field, old value, new value. All of it. In a format that downstream systems can ingest and replay. That's your delta.

Now you might ask, does this really solve our earlier problems?

Absolutely. First, because you're not hammering the source with huge queries, your load stays smooth. None of those scary spikes that happen when your batch jobs kick off at 2 am and block half your transactions.

Second, the data isn't being sent in giant blobs. It's flowing in constantly, in small, digestible chunks. That means fewer network headaches, fewer retries, less infrastructure bloat. You don't need to architect a mega-pipeline just to ship a few updates.

And finally — maybe most importantly — the data in your warehouse is fresh. Not "updated every 24 hours" fresh. Actually fresh. Like, real-time dashboard fresh. That's a huge win for decision-making. You're not looking in the rearview mirror anymore. You're getting data as things happen.

That's the power of CDC.

You're getting an exclusive preview of my latest article! Want to dive deeper? The full story is just a click away: