System Design Case Study

How do you continuously compute top active channels without scanning historical data?

🎯 Design a real-time ranking system that computes top active channels/users for the last 1 hour, updating within seconds of activity changes
Concepts Involved
Stream Processing Redis (Sorted Sets) Kafka Bloom Filters Caching

Given message events across millions of channels, design a system that continuously computes the top active channels/users for the last 1 hour without scanning historical data, updating rankings within seconds of activity changes.

Scope: Real-time top-K computation over a sliding time window · not batch analytics, not historical reporting. The hard question: how do you maintain accurate rankings over a sliding 1-hour window at millions of events/sec without re-scanning all data on every query✗
200K
events / sec ingested
from Kafka message topic
1 hour
sliding window
old activity expires continuously
<5s
ranking freshness
activity → reflected in top-K
Top 100
channels per workspace
approximate is acceptable

Functional Requirements

What the system must do · streaming top-K computation over a sliding window

Must Have (Core)

1. Compute top 100 most active channels per workspace in the last 1 hour
2. Rankings update within 5 seconds of new activity
3. Old activity expires automatically as it falls outside the 1-hour window
4. Support per-workspace and global (cross-workspace) rankings
5. Handle 200K events/sec without scanning historical data
6. Approximate results acceptable (·5% accuracy for top-K)

Out of Scope (this design)

✗ Historical analytics (daily/weekly reports · batch pipeline)
✗ Message content analysis (topic detection, sentiment)
✗ User activity scoring (engagement metrics for product analytics)
✗ Exact counts (approximate top-K is sufficient for ranking)
✗ Custom time windows (fixed at 1 hour for this design)

Non-Functional Requirements

Quality constraints for real-time ranking computation over sliding windows

PropertyTargetWhy It Matters / Design Impact
FreshnessRankings update within 5 seconds of activityStale rankings mislead users. Must reflect recent bursts immediately.
Query latencyP99 <50ms for top-K retrievalRankings are displayed in sidebar · must load faster than the page itself.
Throughput200K+ message events/sec ingestedMust keep up with full message stream without falling behind or dropping events.
AccuracyApproximate acceptable (·5% for ranking position)Exact counts unnecessary · relative ordering is what matters for top-K display.
WindowSliding 1-hour window with automatic expiryOld activity must decay without explicit cleanup. Time-bucketed approach enables this.
ScalabilityPer-workspace rankings + global aggregationEach workspace is independent. Global view aggregates across workspaces on demand.
MemoryBounded regardless of total channel countOnly active channels consume memory. Inactive channels naturally expire from sorted sets.

Scale Estimation

Deriving Redis capacity from event rate and workspace distribution

StepCalculationResult
1200K msgs/sec from Kafka → 200K ZINCRBY ops/sec to Redis200K write ops/sec
260 time buckets · avg 10K active channels/workspace600K ZSET entries
3Memory: 600K entries · 16 bytes~10MB per workspace (trivial)
4ZUNIONSTORE across 60 keys: O(60 · 10K)600K ops every 5s
5Total workspaces: 500K · 10MB~5TB if all active (but most are small)
6Realistic: top 1000 workspaces need dedicated RedisRest share multi-tenant clusters

Architecture Overview

Kafka stream → sliding window aggregation → Redis sorted sets with time-decay scoring → top-K query

INGESTION · Message events from Kafka → sliding window counters Kafka messages topic 200K events/sec Stream Processor extract: workspace_id, channel_id assign to 1-min time bucket ZINCRBY per channel per bucket Redis Sorted Sets active:{workspace}:{bucket} → ZSET member=channel_id, score=msg_count 60 buckets per workspace (1 per minute) TTL Auto-Expiry Each bucket key: EXPIRE 3660s Bucket older than 1hr → auto-deleted No manual cleanup needed Ingestion Latency Kafka → Redis: ~5ms ZINCRBY: ~0.1ms QUERY PATH · "What are the top 100 active channels right now?" Client GET /top-channels Aggregation Service ZUNIONSTORE across 60 buckets or pre-computed every 5s Redis ZUNIONSTORE sum scores across 60 minute-buckets ZREVRANGE 0 99 → top 100 Top 100 Channels ranked by 1hr msg count cached 5s, served to clients Query Latency ZUNIONSTORE: ~10ms ZREVRANGE: ~1ms APPROXIMATE TOP-K · Count-Min Sketch + Min-Heap (for extreme scale) Count-Min Sketch Fixed memory: ~1MB regardless of channel count Increment on every message event Query: approximate count for any channel Error: e·N (configurable accuracy vs. memory) Min-Heap (size K=100) Maintains top-100 channels by count New event: if count > heap min → replace O(log K) per update = O(log 100) = O(7) Memory: 100 entries · 16 bytes = 1.6KB When to Use Which Redis ZSET: <1M channels (exact counts) CMS + Heap: >1M channels (approximate) Hierarchical: per-workspace exact + global approx Most workspaces: Redis ZSET is sufficient

Key Design Decisions

Streaming computation with time-bucketed aggregation · no batch jobs, no historical scans

DecisionChoiceWhyAlternative Rejected
Window StrategyTime-bucketed (1-min buckets · 60)Each minute gets its own Redis sorted set. Union 60 buckets = 1-hour window. Old buckets auto-expire via TTL.Single counter with decay: hard to expire exactly at 1hr. Tumbling window: stale for up to window-size.
StorageRedis Sorted Sets (ZSET)ZINCRBY O(log N) for increment. ZUNIONSTORE for aggregation. ZREVRANGE for top-K. All sub-ms operations.Database: too slow for 200K writes/sec. In-memory only: lost on restart (acceptable for rankings).
ExpiryTTL on bucket keys (3660s)Redis auto-deletes expired buckets. No background cleanup job needed. Sliding window maintained passively.Manual cleanup cron: adds operational complexity. Explicit delete: race conditions with writers.
AccuracyExact per-workspace, approximate globalPer-workspace has <10K channels → Redis ZSET is exact. Global across millionS → CMS + heap is approximate but bounded.Exact global: requires scanning all workspaces. Too expensive for real-time.
Computation ModelStreaming (no batch)Every message event immediately increments the counter. No periodic batch job. Rankings are always fresh within 5s.Batch (every 5 min): stale rankings, spiky compute load. Lambda architecture: unnecessary complexity.
Cross-reference: The event source is the same Kafka messages topic used by the fan-out workers and search indexer. The ranking system is just another consumer group on the same topic · no additional event production needed.

Resilience & Edge Cases

Rankings are ephemeral and rebuildable · the system optimizes for freshness over durability

#Failure / Edge CaseWhat BreaksHow It's Handled
1Redis node failureCurrent bucket data lost · rankings temporarily inaccurateRankings rebuild within 1 minute as new events flow in. Redis Sentinel promotes replica. Stale data acceptable for rankings.
2Kafka consumer lagRankings lag behind real activityAlert on lag >30s. Auto-scale consumers. Rankings are eventually consistent · 30s staleness is acceptable.
3Bot spam in a channel (10K msgs/min)Bot-spammed channel dominates rankingsRate-limit per user at ingestion. Optionally: weight by unique users, not raw message count. Filter known bots.
4ZUNIONSTORE on 60 keys is slowQuery latency spikes during aggregationPre-compute union every 5s into a result key. Clients read from cached result. ZUNIONSTORE runs in background.
5Workspace with 100K+ channelsZSET per bucket becomes largeOnly track channels with >0 messages in the bucket. Most channels are inactive · ZSET stays small. Worst case: shard across Redis nodes.
Design philosophy: Rankings are a derived, ephemeral view. They can be rebuilt from the Kafka topic at any time. A few seconds of staleness or inaccuracy is invisible to users. This allows aggressive optimization for write throughput over read consistency.

Tech Stack & Tradeoffs

Minimal stack · streaming aggregation with Redis as the computation engine

ComponentTechnologyWhy ThisWhy Not X
Event SourceKafka (messages topic)Same topic as message delivery. Ranking is just another consumer group. No additional event production needed.Separate event stream: duplicates data, adds complexity. CDC from DB: higher latency.
Stream ProcessorCustom consumer (Go/Java)Simple logic: extract workspace+channel, ZINCRBY Redis. No need for Flink/Spark complexity.Flink: powerful but operationally heavy for simple counting. Spark Streaming: batch-oriented, higher latency.
Aggregation StoreRedis Sorted SetsZINCRBY for counting, ZUNIONSTORE for window aggregation, ZREVRANGE for top-K. All O(log N) or better.TimescaleDB: too slow for 200K writes/sec. ClickHouse: batch-oriented, not real-time enough.
Approximate CountingCount-Min Sketch (for global)Fixed memory (~1MB) regardless of cardinality. Bounded error. Perfect for "approximate top-K" requirement.Exact counting at global scale: requires O(channels) memory. HyperLogLog: counts uniques, not frequencies.
Result CacheRedis key with 5s TTLPre-computed top-100 result. Clients read cached result. Background job refreshes every 5s.Compute on every query: ZUNIONSTORE on 60 keys per request is wasteful. CDN: too aggressive caching.
Cross-reference: This system consumes from the same Kafka pipeline that powers message delivery, search indexing, and offline delivery. It's a read-only consumer that never affects the write path.

Interview Cheat Sheet

The 5 things an interviewer wants to hear about real-time ranking systems

? Time-Bucketed Sliding Window (60 · 1-min buckets)
Split the 1-hour window into 60 one-minute buckets. Each bucket is a Redis sorted set. ZINCRBY on every message. Union all 60 buckets for the full window. Old buckets auto-expire via Redis TTL · no cleanup job needed.
? Redis Sorted Sets for O(log N) Ranking
ZINCRBY for counting (O(log N)), ZUNIONSTORE for window aggregation (O(N·K)), ZREVRANGE for top-K (O(log N + K)). All operations are sub-ms. Redis is the computation engine, not just a cache.
? Count-Min Sketch + Min-Heap for Extreme Scale
When exact counting is too expensive (millions of channels globally): CMS gives approximate frequency in O(1) with fixed memory. Min-heap of size K maintains the top-K candidates. Together: O(1) per event, O(K) memory for results.
? Streaming, Not Batch · Always Fresh
Every message event immediately increments the counter via Kafka consumer. No periodic batch job. Rankings reflect activity within 5 seconds. Pre-compute the union result every 5s into a cached key for fast reads.
? Hierarchical: Per-Workspace Exact → Global Approximate
Per-workspace rankings use exact Redis ZSETs (most workspaces have <10K channels). Global cross-workspace rankings use approximate structures (CMS + heap). This gives exact results where users care most and approximate where scale demands it.
One-liner answer: "Kafka consumer increments Redis sorted sets in 1-minute time buckets, ZUNIONSTORE across 60 buckets for the sliding 1-hour window, TTL auto-expires old buckets, pre-computed top-100 result cached every 5 seconds, with Count-Min Sketch + min-heap for approximate global rankings at extreme scale."