System Design Case Study

Design a real-time in-app notification feed with badge counts and WebSocket push

🎯 Design a real-time in-app notification feed with unread badge counts, WebSocket push for instant updates, Redis sorted set for feed storage, and cursor-based pagination for 500M users
Concepts Involved
WebSocket Redis Sorted Set Pub/Sub Fan-out Cursor-Based Pagination Connection Sharding

Problem Statement

How do you build the notification bell/badge that shows unread count, displays a chronological feed, supports mark-as-read, and pushes new notifications in real-time via WebSocket to 10M concurrent connections?

Scope: In-app notification feed (the bell icon), real-time badge count updates via WebSocket, unread counter with atomic operations, cursor-based feed pagination, mark-as-read mechanics. Not covering push/email/SMS delivery or notification content generation.
500M
total users
each with notification feed
10M
concurrent WebSocket connections
real-time badge push
<50ms
feed read latency
Redis sorted set lookup
200ms
WebSocket push latency
new notification → badge update

Functional Requirements

What the notification feed system must do · core feed behaviours

Must Have (Core)

1. Notification feed · chronological list of notifications with cursor-based pagination
2. Unread badge count · accurate count displayed on bell icon, atomically incremented/decremented
3. Real-time push via WebSocket · new notifications push badge update + feed item to connected clients
4. Mark as read · single notification or bulk mark-all-as-read, decrement unread counter
5. Feed truncation · keep last 200 notifications per user (older ones archived)
6. Multi-device sync · mark-as-read on one device reflects on all connected devices

Out of Scope (this design)

✗ Notification generation/aggregation (upstream service)
✗ Push/email/SMS delivery (separate channel workers)
✗ Notification preferences (opt-in/opt-out management)
✗ Rich media in notifications (images, action buttons)
✗ Notification grouping UI (client-side rendering logic)

Non-Functional Requirements

Quality constraints shaping the notification feed architecture

PropertyTargetWhy It Matters / Design Impact
Feed Read Latency<50ms P99Users expect instant feed display on bell click. Redis ZREVRANGEBYSCORE provides sub-ms response.
WebSocket Push<200ms end-to-endNew notification must update badge within 200ms of creation. Requires efficient connection routing.
Concurrent Connections10M WebSocketEach connection holds ~10KB state. 10M × 10KB = 100GB total across WebSocket fleet.
Unread Counter ConsistencyStrongly consistentBadge count must be accurate. Use Redis INCR/DECR atomic operations. No stale reads.
Multi-Device Sync<1s propagationMark-as-read on phone must update badge on web within 1 second.
Availability99.99%Notification feed is always-visible UI element. Degraded mode: show cached feed if Redis unavailable.
Feed Size200 items maxBounded feed prevents unbounded Redis memory growth. ZREMRANGEBYRANK trims on each write.
Key tension: Consistency vs. Availability of unread count. Users are sensitive to incorrect badge counts (showing 3 when 0 unread). Solution: Redis atomic counter as source of truth, with WebSocket push propagating updates across all devices within 1s.

Scale Estimation

Derive infrastructure sizing from given constraints. Show this reasoning in an interview.

Given: 500M users · 10M concurrent WS connections · 2B notifications written/day · 200 feed items/user max
StepWhat to DeriveCalculationResultDesign Decision
1 Feed writes/sec 2B / 86,400s ~23K/sec ZADD to per-user sorted set. Redis cluster handles this easily.
2 WebSocket push messages/sec 23K new notifications/sec × avg 1.2 connected devices/user ~28K pushes/sec WebSocket gateway routes by user_id → connection shard.
3 Redis memory (all feeds) 500M users × 200 items × 128 bytes avg ~12.8TB total But only active users (20%) in Redis. Cold users on disk. ~2.5TB hot.
4 Hot user feeds in Redis 100M active users × 200 items × 128B ~2.5TB Redis cluster: 256 shards × 10GB each. Shard by user_id hash.
5 WebSocket server fleet 10M connections / 100K connections per server ~100 WS servers Each server: 64GB RAM, handles 100K persistent connections.
6 Feed read QPS 500M DAU × 5 feed opens/day / 86,400s ~29K reads/sec ZREVRANGEBYSCORE with cursor. Sub-ms per read from Redis.
7 Unread counter storage 500M users × 8 bytes (integer) ~4GB Separate Redis cluster for counters. Atomic INCR/DECR.
8 Mark-as-read QPS ~50% of feed opens trigger mark-as-read = ~14.5K/sec ~14.5K/sec Batch mark-all-as-read: single SET unread:{user_id} 0
Interview tip: The key insight is that not all 500M users need feeds in Redis. Only active users (DAU ~100M) need hot feeds. Cold users' feeds are loaded from persistent storage (DynamoDB/Cassandra) into Redis on first access. This reduces Redis memory from 12.8TB to ~2.5TB.

APIs

Notification feed endpoints · REST for reads/writes, WebSocket for real-time push

GET /v1/notifications/{user_id}/feed
Retrieve paginated notification feed with cursor-based pagination
GET /v1/notifications/u_abc123/feed?limit=20&cursor=MTcxMDQ5NTAwMA==

Response 200:
{
  "notifications": [
    {
      "id": "n_001", "type": "like_aggregated",
      "title": "Jane and 4 others liked your post",
      "body": "My vacation photos...",
      "actors": [{"id": "u_456", "name": "Jane", "avatar": "..."}],
      "entity": {"type": "post", "id": "p_789"},
      "read": false,
      "created_at": "2024-03-15T10:30:00Z"
    }
  ],
  "next_cursor": "MTcxMDQ5NDAwMA==",
  "has_more": true,
  "unread_count": 7
}
GET /v1/notifications/{user_id}/unread-count
Get current unread badge count (fast path for badge display)
GET /v1/notifications/u_abc123/unread-count

Response 200: { "unread_count": 7, "last_updated": "2024-03-15T10:30:00Z" }
PUT /v1/notifications/{user_id}/read
Mark notifications as read (single or bulk)
PUT /v1/notifications/u_abc123/read
{
  "notification_ids": ["n_001", "n_002"],  // specific IDs, or...
  "mark_all": false,                        // true = mark all as read
  "read_before": null                       // mark all before timestamp
}

Response 200: { "marked_count": 2, "unread_count": 5 }
WebSocket /ws/notifications/{user_id}
Real-time push channel for badge updates and new notifications
// Server → Client messages:
{ "type": "badge_update", "unread_count": 8 }
{ "type": "new_notification", "notification": { "id": "n_003", "title": "...", ... } }
{ "type": "read_sync", "notification_ids": ["n_001"], "unread_count": 7 }

// Client → Server messages:
{ "type": "ack", "notification_id": "n_003" }
{ "type": "mark_read", "notification_ids": ["n_003"] }

Data Model

Redis structures and persistent storage for the notification feed

Notification Feed (Redis Sorted Set)

// Per-user feed — sorted by timestamp (score = created_at unix ms)
// Key: feed:{user_id}
ZADD feed:u_abc123 1710495000000 "n_001"
ZADD feed:u_abc123 1710494000000 "n_002"

// Trim to max 200 entries on each write
ZREMRANGEBYRANK feed:u_abc123 0 -201

// Pagination: cursor-based using score (timestamp)
ZREVRANGEBYSCORE feed:u_abc123 {cursor_ts} -inf LIMIT 0 20

Notification Details (Redis Hash)

// Notification content stored separately (referenced by ID from feed)
HSET notif:n_001
  "type"        "like_aggregated"
  "title"       "Jane and 4 others liked your post"
  "body"        "My vacation photos..."
  "actors"      "[{\"id\":\"u_456\",\"name\":\"Jane\"}]"
  "entity_type" "post"
  "entity_id"   "p_789"
  "read"        "0"
  "created_at"  "1710495000000"
  "collapse_key" "like:p_789:u_abc123"

EXPIRE notif:n_001 2592000  // 30 day TTL

Unread Counter (Redis)

// Atomic counter per user
SET unread:u_abc123 7

// On new notification:
INCR unread:u_abc123

// On mark-as-read (single):
DECR unread:u_abc123

// On mark-all-as-read:
SET unread:u_abc123 0

// Safety: counter can't go below 0
// Use Lua script: if tonumber(redis.call('GET', key)) > 0 then redis.call('DECR', key) end

WebSocket Connection Registry (Redis)

// Track which WS server holds which user's connection
// Key: ws_conn:{user_id}, Value: ws_server_id
HSET ws_conn:u_abc123 "device_web" "ws-server-042"
HSET ws_conn:u_abc123 "device_mobile" "ws-server-017"

EXPIRE ws_conn:u_abc123 300  // 5 min TTL, refreshed by heartbeat

// Reverse index: which users are on which server (for server shutdown drain)
SADD ws_server:ws-server-042 "u_abc123" "u_def456" ...

Persistent Store (DynamoDB — cold storage)

{
  "user_id":         "u_abc123",       // partition key
  "notification_id": "n_001",          // sort key
  "type":            "like_aggregated",
  "title":           "Jane and 4 others liked your post",
  "actors":          [{"id": "u_456", "name": "Jane"}],
  "entity":          {"type": "post", "id": "p_789"},
  "read":            false,
  "created_at":      "2024-03-15T10:30:00Z",
  "ttl":             1718131200        // 90-day auto-delete
}

Architecture Overview

Write Path: Notification → Redis Feed + Counter → Pub/Sub → WebSocket Push
Read Path: Client → API → Redis ZREVRANGEBYSCORE → Paginated Response

WRITE PATH · New notification arrives from upstream service Notification Service (upstream) Feed Writer (Kafka consumer) DynamoDB (persistent write-through) REDIS CLUSTER · Feed storage + unread counters + connection registry Feed ZSET (per user) ZADD + ZREMRANGEBYRANK (trim 200) 256 shards · ~2.5TB hot data Unread Counter INCR unread:{user_id} Atomic · ~4GB cluster Notif Detail Hash HSET notif:{id} (content) 30-day TTL auto-expire WS Connection Map user → ws_server_id Heartbeat refresh 5min PUB/SUB ROUTING · Route push message to correct WebSocket server Redis Pub/Sub Channel PUBLISH ws_push:{server_id} {payload} Connection Lookup HGET ws_conn:{user_id} → target server(s) Multi-Device Fan-out Send to ALL connected devices for user WEBSOCKET GATEWAY · 100 servers × 100K connections each WS Server 1 100K connections WS Server 2 100K connections ... WS Server 100 100K connections Sticky LB (by user_id) Consistent hash → same server READ PATH · Client requests feed / unread count Feed API GET /feed → ZREVRANGEBYSCORE Cursor = last item timestamp Badge API GET /unread-count → GET unread:{id} Single Redis GET · O(1) Mark-Read API DECR unread + HSET read=1 + publish read_sync to other devices Cold Load (cache miss) DynamoDB → Redis hydration On first access for cold user KEY INSIGHT Connection registry maps user → server. No broadcast needed. Direct routing to correct WS server.

Key Design Decisions

Tradeoffs made in this architecture and their rationale

DecisionOptions ConsideredChosenWhy
Feed Storage List (LPUSH), Sorted Set (ZADD), Stream (XADD) Redis Sorted Set Score = timestamp enables cursor-based pagination via ZREVRANGEBYSCORE. O(log N) inserts. ZREMRANGEBYRANK for bounded size.
WebSocket Routing Broadcast to all WS servers, connection registry lookup, consistent hashing Connection registry + direct routing Broadcasting wastes 99% of messages (user on 1 of 100 servers). Registry lookup is O(1) and routes directly to correct server.
Unread Counter Count from feed (ZCOUNT), separate counter, client-side tracking Separate atomic Redis counter ZCOUNT is O(log N) on every badge render. Separate counter is O(1) GET. Trade off slight drift risk for massive read performance.
Pagination Offset-based, cursor-based (timestamp), keyset Cursor-based (timestamp score) Offset breaks when new items are added mid-page. Timestamp cursor is stable — "give me 20 items before this timestamp" always works correctly.
Multi-Device Sync Polling, WebSocket push, push notification WebSocket push to all connected devices Mark-as-read on phone → publish via Redis Pub/Sub → route to all WS servers holding user connections → instant badge update on web.
Hot/Cold Feed Strategy All feeds in Redis, LRU eviction, explicit hot/cold split Explicit: active users in Redis, cold in DynamoDB Only 20% of users are active daily. Loading all 500M feeds into Redis wastes 10TB. Load on first access, evict after 7 days of inactivity.
Why connection registry over broadcast: With 100 WS servers, broadcasting "new notification for user X" to all 100 servers means 99 servers discard it (user is only on 1). Connection registry: HGET ws_conn:{user_id} → "ws-server-042". Publish directly to that server's channel. 100x more efficient.
Cursor-based pagination with sorted sets: Client passes cursor = last seen timestamp. Server does ZREVRANGEBYSCORE feed:{user_id} (cursor-1) -inf LIMIT 0 20. New notifications added AFTER the cursor don't affect pagination. Client can also pull "new items since cursor" with ZRANGEBYSCORE feed:{user_id} cursor +inf.
Unread counter drift protection: The counter can drift from reality (race conditions, failed decrements). Periodic reconciliation job: ZCOUNT feed:{user_id} -inf +inf WHERE read=false vs GET unread:{user_id}. If they differ, SET counter to ZCOUNT result. Run hourly for active users.
Graceful WebSocket server shutdown: Before terminating a WS server, drain connections: notify clients to reconnect (send "reconnect" frame), update connection registry (remove entries), wait 30s for clients to re-establish on other servers. LB health check marks draining server unhealthy immediately.

Resilience & Edge Cases

How the notification feed system handles failures gracefully

ScenarioProblemSolutionRecovery
WebSocket Server Crash Server dies, 100K connections drop. Clients reconnect to other servers. Client auto-reconnect with exponential backoff. On reconnect: register new connection, fetch missed notifications (diff feed since last_seen_ts). <5s client reconnect
Redis Feed Cluster Down Feed sorted sets unavailable. Can't read or write feed. Fallback to DynamoDB direct read (slower, ~50ms → ~200ms). Queue writes for replay when Redis recovers. Badge count from separate counter cluster. Degraded (higher latency reads)
Counter Drift Badge shows 5, but user only has 3 unread notifications. Hourly reconciliation job for active users. On mark-all-as-read: SET counter to 0 (corrects any drift). Client can also verify on feed load. Self-healing within 1 hour
Notification Flood Viral post generates 10K notifications for one user in 1 minute. Per-user write rate limit: max 10 feed writes/minute. Excess notifications are aggregated into batch entries. Counter still increments accurately. Automatic rate throttling
Stale Connection Registry User disconnected but registry still points to old server (no heartbeat cleanup). 5-minute TTL on registry entries. Heartbeat refreshes TTL. If push fails (user not found on target server), remove registry entry. Next connection re-registers. <5min auto-cleanup
Cold User Activation User inactive for months returns. Feed not in Redis. Cache miss triggers DynamoDB query → hydrate Redis feed. Takes ~200ms for first load. Subsequent reads are sub-ms from Redis. Pre-warm for predicted re-engagers. 200ms first-load penalty

Interview Cheat Sheet

5 key points to nail a real-time notification feed interview

1. Redis Sorted Set for Feed + Cursor Pagination
Each user has a sorted set (score = timestamp). ZADD on new notification. ZREVRANGEBYSCORE for reverse-chronological pagination. Cursor = last item's timestamp. ZREMRANGEBYRANK keeps feed bounded to 200 items. Sub-ms reads, O(log N) writes. The sorted set IS the feed.
2. Connection Registry for Direct WebSocket Routing
Don't broadcast to all WS servers. Maintain a Redis hash mapping user_id → ws_server_id. On new notification: lookup target server, PUBLISH directly to that server's channel. Server finds the user's socket in its local connection table. O(1) routing, no wasted messages.
3. Atomic Unread Counter (Separate from Feed)
Don't compute unread count by scanning the feed (O(N) on every badge render). Maintain a separate Redis counter: INCR on new notification, DECR on mark-as-read, SET 0 on mark-all-as-read. O(1) badge reads. Reconciliation job fixes drift hourly.
4. Multi-Device Sync via Pub/Sub
Mark-as-read on phone must update badge on web instantly. Solution: on mark-read, after DECR counter, PUBLISH a "read_sync" message to ALL servers holding user's connections (from registry). Each server pushes badge update via WebSocket. All devices converge within 200ms.
5. Hot/Cold Split — Only Active Users in Redis
500M users but only 100M DAU. Keeping all feeds in Redis = 12.8TB (wasteful). Only active users' feeds live in Redis (~2.5TB). Cold users' feeds are in DynamoDB. On first access after inactivity, hydrate from DynamoDB → Redis (200ms penalty once). Evict after 7 days of no access.