Stream DB changes from the transaction log to improve
real-time updates (propagate changes),
decoupling (no app code changes), and
integration (sync downstream systems),
with trade-offs in
consistency (event lag),
ordering (out-of-order events), and
operational complexity (setup, monitoring).
▸ CDC Pipeline — How It Works
▸ Why CDC? The Dual-Write Problem
❌ Dual Write (Dangerous)
✓ CDC (Safe)
▸ CDC Methods
Method
How
Latency
Impact on Source
Log-Based
Read WAL/binlog (Debezium)
~ms
Zero — reads log, not tables
Query-Based
Poll with WHERE updated_at > ?
~sec-min
Load — queries hit DB
Trigger-Based
DB triggers write to shadow table
~ms
High — triggers on every write
Guarantees:At-least-once delivery (consumers must be idempotent). Ordering per table. No dual-write problem — single source of truth is the DB, everything else derived from its log.
Real-world:LinkedIn — 100+ Kafka topics via CDC. Shopify — real-time inventory sync. Airbnb — Debezium for search index updates. Stripe — outbox + CDC for payment events.
CDC Tools:Debezium — open-source, log-based, Kafka Connect plugin (most popular). AWS DMS — managed, serverless, AWS-native migrations + replication. Fivetran — SaaS, 200+ connectors, zero-config ELT to warehouses. Airbyte — open-source Fivetran alternative, 300+ connectors, self-hosted or cloud. All log-based = zero impact on source DB.
ETL / ELT
ETL: Transform before loading (traditional). ELT: Load raw, transform in warehouse (modern).
Process data in real-time as it arrives — event by event, no waiting
Framework
Model
Real-World Example
Apache Flink
True streaming, exactly-once, stateful
Uber — surge pricing: count ride requests per zone in 5-min sliding window → price multiplier in real-time
Kafka Streams
Library (no cluster), Kafka ecosystem
LinkedIn — "Who viewed your profile" notifications: join view events with user profiles, emit within seconds
Spark Streaming
Micro-batch (100ms–sec intervals)
Netflix — real-time viewing metrics: aggregate play/pause/skip events per title for trending dashboard
Key Concepts:Windowing — tumbling (fixed, non-overlapping: "count per 5 min"), sliding (overlapping: "last 10 min, updated every 1 min"), session (gap-based: "user activity until 30 min idle"). Watermarks — handle late-arriving events ("allow 5 sec late data"). Exactly-once — checkpointing + idempotent sinks. State — RocksDB in Flink for keyed state (e.g., running count per user).
Batch Processing
Process large datasets in scheduled jobs — collect first, process later
MapReduce → Spark
Real-World Batch Examples
▸Netflix — nightly Spark job processes PBs of viewing data → "Because you watched" recommendations ▸Spotify — daily batch: aggregate all listening history → generate "Discover Weekly" playlist ▸Uber — end-of-day batch: calculate driver earnings, trip summaries, tax reports ▸Banks — nightly batch: reconcile all transactions, generate statements, fraud reports
Aspect
Stream Processing
Batch Processing
When
As data arrives (continuous)
Scheduled (hourly/daily/weekly)
Latency
ms – seconds
minutes – hours
Data
Unbounded (infinite stream)
Bounded (fixed dataset)
State
In-memory (RocksDB, checkpoints)
Disk (HDFS, S3)
Use Case
Fraud detection, surge pricing, alerts
Reports, ML training, ETL, analytics
Tools
Flink, Kafka Streams, Spark Streaming
Spark, MapReduce, Hive, Presto
Lambda Architecture: Run both — batch for accuracy (recompute everything), stream for speed (approximate, real-time). Merge results in serving layer. Kappa Architecture:Stream only — replay Kafka log for reprocessing instead of separate batch. Simpler but needs durable stream (Kafka retention).
Data Warehouse
OLAP — columnar storage, fast aggregations. BigQuery, Snowflake, ClickHouse
Guarantee:Columnar storage reads only needed columns — SUM(revenue) skips all other columns. 10:1 compression. OLTP — Runs the business (transactional). OLAP — Analyzes the business (analytical).
Data Lakes & Lakehouse
Raw files on S3/GCS. Lakehouse = lake (cheap) + warehouse (ACID). Delta Lake, Iceberg, Hudi add ACID + time travel.