System Design Case Study

Design notification preferences with topic subscriptions, quiet hours, and frequency caps

🎯 Design a preference system where users opt-in/out per topic per channel, set timezone-aware quiet hours, and the system enforces frequency caps (max 5 marketing/day) — all checked in <1ms BEFORE sending
Concepts Involved
Bloom Filter Sliding Window Distributed Cache GDPR Compliance Timezone Handling

Problem Statement

Users opt-in/out per topic per channel, set quiet hours (timezone-aware), and the system enforces frequency caps (max 5 marketing notifications/day). All preference checks happen BEFORE sending — on the critical path of every notification. With 500M users and 50 preference entries each, how do you check 25B preference records in <1ms per notification?

Scope: Preference storage and lookup, Bloom filter for fast rejection, quiet hours with timezone handling, sliding window frequency caps. Integrates with notification service channel router as the gating layer before fan-out.
25B
preference records
500M users × 50 entries each
<1ms
preference check latency
on every notification send path
<5s
opt-out propagation
real-time unsubscribe effect
70%
fast-rejected by Bloom filter
never hits Redis for unsubscribed

Functional Requirements

What the preference and frequency cap system must do

Must Have (Core)

1. Per-topic per-channel opt-in/out · user subscribes/unsubscribes to "promotions" on "push" independently from "email"
2. Quiet hours · timezone-aware suppression window (e.g., 10pm-8am user-local time)
3. Frequency caps · max N notifications per category per time window (sliding window counter)
4. Bloom filter fast path · reject 70% of "definitely unsubscribed" checks without Redis lookup
5. Real-time opt-out propagation · unsubscribe takes effect within 5s across all workers
6. GDPR unsubscribe-all · one-click disable all non-essential notifications

Out of Scope (this design)

✗ Preference UI (frontend settings page)
✗ Notification content (templates, rendering)
✗ Delivery mechanics (APNs/FCM/Twilio integration)
✗ Analytics (opt-out rates, engagement metrics)
✗ A/B testing of notification frequency
✗ Preference migration from legacy systems

Non-Functional Requirements

Quality constraints for the preference checking layer

PropertyTargetWhy It Matters / Design Impact
Check Latency<1ms per notificationOn the critical send path. At 58K notifications/sec, even 5ms adds 290 worker-seconds/sec. Must be sub-millisecond.
Opt-out Propagation<5s from user actionGDPR/CAN-SPAM compliance. User clicks unsubscribe → no more notifications within 5s. Pub/sub cache invalidation.
False Positive Rate<0.1% Bloom filterBloom filter may say "possibly subscribed" when actually unsubscribed (false positive → falls through to Redis). Never says "unsubscribed" when actually subscribed (no false negatives).
Storage Efficiency25B records in-memoryCan't store all 25B preferences in Redis. Bloom filter compresses to ~3GB for 500M users. Redis stores only active/hot preferences.
GDPR ComplianceImmediate effectUnsubscribe-all must propagate globally. No "we'll stop in 24 hours" — must be immediate. Bloom filter rebuilt async.
Timezone AccuracyCorrect to user's offsetQuiet hours must respect DST transitions. Store timezone name (not offset). Calculate at check time.
Key tension: Speed vs. Freshness. Bloom filter gives <0.1ms checks but can't be updated in real-time (rebuild takes minutes). Redis gives real-time freshness but costs 1-2ms per lookup. Solution: Bloom filter for fast rejection of 70% traffic, Redis fallback for the 30% that Bloom says "maybe subscribed".

Scale Estimation

Sizing the preference infrastructure for 500M users

Given: 500M users · 50 preference entries/user · 10 topics × 4 channels + quiet hours + freq caps · 58K checks/sec
StepWhat to DeriveCalculationResultDesign Decision
1 Total preference records 500M users × (10 topics × 4 channels + 1 quiet_hours + 10 freq_caps) 25.5B records Can't fit in memory. Need tiered approach: Bloom + Redis + DB.
2 Bloom filter size (unsubscribes) ~2B unsubscribed entries × 10 bits/entry (0.1% FPR) ~2.5GB Fits in worker memory. Loaded on startup. Rebuilt every 15 min.
3 Checks/sec hitting Redis (post-Bloom) 58K × 30% pass-through ~17.4K Redis lookups/sec Redis cluster handles 100K+ ops/sec easily. Well within capacity.
4 Redis memory for hot preferences 100M active users × 50 entries × 32 bytes ~160GB Redis cluster (32 shards). LRU eviction for inactive users. Reload from DB on miss.
5 Frequency cap counters 100M active users × 5 categories × 32 bytes ~16GB Redis with TTL matching window size. Sliding window via sorted set or INCR+EXPIRE.
6 Quiet hours check CPU 58K/sec × timezone calculation (~10μs each) ~0.58 CPU cores Negligible. Cache timezone offset per user (recalc on DST change).
7 Bloom filter rebuild time 2B entries × 10 hash operations / 1M ops/sec ~33 minutes Background rebuild every 15 min (overlapping). Hot-swap on completion. Old filter serves during rebuild.
Interview tip: The Bloom filter is the key optimization. Without it, every notification requires a Redis round-trip. With it, 70% of notifications are rejected in <0.1ms (in-process memory). Only the 30% that pass the Bloom filter (possibly subscribed) hit Redis. This reduces Redis load from 58K/sec to 17K/sec — a 3.3× reduction.

APIs

Preference management and GDPR compliance endpoints

PUT /v1/users/{id}/preferences
Update user notification preferences (topic + channel granularity)
{
  "preferences": [
    { "topic": "promotions", "channel": "push", "enabled": false },
    { "topic": "promotions", "channel": "email", "enabled": true },
    { "topic": "order_updates", "channel": "push", "enabled": true },
    { "topic": "order_updates", "channel": "sms", "enabled": true }
  ],
  "quiet_hours": {
    "enabled": true,
    "start": "22:00",
    "end": "08:00",
    "timezone": "America/New_York"
  },
  "frequency_caps": [
    { "category": "marketing", "max_count": 5, "window": "24h" },
    { "category": "social", "max_count": 20, "window": "24h" }
  ]
}
Response: 200 OK · Propagates to cache within 5s via pub/sub invalidation
GET /v1/users/{id}/preferences
Retrieve all preferences for a user (used by settings UI)
GET /v1/users/u_abc123/preferences

Response 200:
{
  "user_id": "u_abc123",
  "preferences": [...],
  "quiet_hours": { "enabled": true, "start": "22:00", "end": "08:00", "timezone": "America/New_York" },
  "frequency_caps": [...],
  "global_unsubscribe": false,
  "updated_at": "2024-01-15T10:00:00Z"
}
POST /v1/preferences/unsubscribe-all
GDPR one-click unsubscribe — disables all non-essential notifications immediately
{
  "user_id": "u_abc123",
  "reason": "user_requested",          // user_requested | gdpr_request | account_deletion
  "retain_essential": true              // keep security/auth notifications
}

Response 200:
{
  "user_id": "u_abc123",
  "global_unsubscribe": true,
  "essential_only": true,
  "effective_at": "2024-01-15T10:00:01Z",   // <1s from request
  "topics_disabled": 9,
  "channels_disabled": 36
}

Data Model

Preference storage: Bloom filter + Redis + persistent DB

Redis: Topic-Channel Preferences

// Per-user per-topic per-channel preference
// Key: preferences:{user_id}:{topic}:{channel}
SET preferences:u_abc123:promotions:push "false"
SET preferences:u_abc123:promotions:email "true"
SET preferences:u_abc123:order_updates:push "true"

// Alternatively, hash per user (fewer keys):
HSET prefs:u_abc123 "promotions:push" "0" "promotions:email" "1" "order_updates:push" "1"

Redis: Quiet Hours

// User's quiet hours configuration
HSET quiet_hours:u_abc123
  enabled    "true"
  start      "22:00"
  end        "08:00"
  timezone   "America/New_York"
  utc_offset "-05:00"           // cached, recalculated on DST change

Redis: Frequency Caps (Sliding Window)

// Sliding window counter using sorted set
// Key: freq_cap:{user_id}:{category}:{window}
// Score = timestamp, Member = notification_id

ZADD freq_cap:u_abc123:marketing:24h 1705312200 "n_001"
ZADD freq_cap:u_abc123:marketing:24h 1705315800 "n_002"

// Check count in window:
ZCOUNT freq_cap:u_abc123:marketing:24h (now - 86400) +inf
// If count >= 5, suppress notification

// Cleanup expired entries:
ZREMRANGEBYSCORE freq_cap:u_abc123:marketing:24h 0 (now - 86400)

Bloom Filter Structure (In-Memory)

// Bloom filter encodes "unsubscribed" entries
// Key format: "{user_id}:{topic}:{channel}"
// Inserted when user unsubscribes

bloom.add("u_abc123:promotions:push")    // user unsubscribed from push promotions

// At check time:
if bloom.contains("u_abc123:promotions:push"):
    return SUPPRESSED   // definitely unsubscribed (no false negatives)
else:
    // might be subscribed — check Redis for exact answer
    redis.get("preferences:u_abc123:promotions:push")

// Bloom filter properties:
//   Size: 2.5GB for 2B entries
//   Hash functions: 7 (optimal for 0.1% FPR)
//   Rebuild: every 15 min from DB snapshot

Architecture Overview

Notification arrives → Bloom filter → Redis preferences → Quiet hours → Frequency cap → Allow/Suppress

INCOMING NOTIFICATION · From channel router (58K/sec) user_id + topic + channel + category → Must check preferences before delivery LAYER 1: BLOOM FILTER · In-process memory (~0.05ms) bloom.contains("{user_id}:{topic}:{channel}") 2.5GB filter · 7 hash functions · 0.1% false positive rate 70% FAST REJECT ✓ Definitely unsubscribed → suppress 30% PASS THROUGH Maybe subscribed → check Redis LAYER 2: REDIS PREFERENCE LOOKUP · Exact check (~0.5ms) HGET prefs:{user_id} "{topic}:{channel}" Exact subscription status Cache miss → load from DynamoDB Async populate cache on first access LAYER 3: QUIET HOURS CHECK · Timezone-aware (~0.1ms) Convert UTC now → user local time (timezone lookup) If local_time ∈ [quiet_start, quiet_end] → suppress (queue for later) DST handling: store timezone NAME not offset "America/New_York" auto-adjusts for EST/EDT transitions LAYER 4: FREQUENCY CAP · Sliding window counter (~0.3ms) ZCOUNT freq_cap:{user}:{cat}:{window} (now-window) +inf If count ≥ max_count → suppress (marketing: max 5/day) On ALLOW: ZADD + set expire on key for cleanup Atomic: check count + increment in one Lua script ALL CHECKS PASS → ALLOW DELIVERY · Total check time: <1ms (Bloom) or <2ms (Bloom miss + Redis)

Key Design Decisions

Tradeoffs for the preference checking layer

DecisionOptions ConsideredChosenWhy
Fast Rejection Redis-only, local cache, Bloom filter Bloom filter + Redis fallback Bloom rejects 70% in ~0.05ms (in-process). Only 30% hit Redis. Reduces Redis load 3.3× and cuts P99 latency significantly.
Frequency Cap Algorithm Fixed window, sliding window (ZSET), sliding window log Redis Sorted Set sliding window True sliding window (no edge-of-window spikes). ZCOUNT is O(log N + M) but M ≤ 20 typically. Auto-cleanup via ZREMRANGEBYSCORE.
Quiet Hours Timezone Store UTC offset, store timezone name, client-side check Store timezone name (IANA) "America/New_York" handles DST automatically. Offset-based breaks twice a year. Server-side check ensures enforcement.
Opt-out Propagation DB poll, cache TTL, pub/sub invalidation Redis pub/sub + Bloom rebuild Pub/sub gives <5s propagation to all workers. Bloom filter can't update atomically — rebuilt every 15 min. Gap covered by Redis lookup on Bloom miss.
GDPR Unsubscribe Soft delete preferences, global flag, delete all records Global flag + async cleanup Setting a global flag is instant (1 Redis SET). All preference checks first check the global flag. Async job cleans up individual preferences later.
Bloom filter insight: We encode UNSUBSCRIBES in the Bloom filter, not subscribes. Why? Because if the Bloom filter says "this key exists" (unsubscribed), we can suppress immediately with zero false negatives. If it says "doesn't exist" (possibly subscribed), we fall through to Redis for the exact answer. This way, the fast path handles the most common suppression case.
Quiet hours edge case: What if a notification is suppressed during quiet hours? Don't drop it. Queue it for delivery at the end of quiet hours (8:00am user local time). This requires a delayed delivery queue indexed by user timezone. The notification is "parked" and released when quiet hours end.
Frequency cap race condition: Two workers check cap simultaneously, both see count=4 (below max 5), both send. User gets 6 notifications instead of 5. Solution: atomic Lua script that does ZCOUNT + ZADD in one operation. If count ≥ max, return SUPPRESSED without incrementing.
GDPR "right to be forgotten": Unsubscribe-all sets a global flag checked FIRST in the preference pipeline (before even the Bloom filter). This guarantees immediate effect. The flag is propagated via pub/sub to all worker instances within 1s. No notification can "slip through" after the flag is set.

Resilience & Edge Cases

Failure modes in the preference checking layer

ScenarioProblemSolutionRecovery
Redis Down Preference cache unavailable — can't check subscriptions. Fail-open for transactional (still send OTP). Fail-closed for marketing (suppress if can't verify). Circuit breaker on Redis calls. Graceful degradation
Bloom Filter Stale User unsubscribed 10 min ago, Bloom filter not yet rebuilt. Bloom filter only fast-rejects. For "maybe subscribed" (pass-through), Redis always has the latest. Stale Bloom = more Redis hits, not incorrect sends. Correctness maintained (latency degrades)
DST Transition Clock jumps 1 hour. Quiet hours window shifts unexpectedly. Use IANA timezone names. Recalculate offset at check time (not cached offset). Libraries handle DST automatically. Automatic via timezone library
Frequency Cap Burst Event triggers 10 notifications simultaneously for same user. All check before any increment. Atomic Lua script: ZCOUNT + conditional ZADD. Only one notification per Lua execution can pass. Others rejected atomically. Atomic enforcement
New User (Cold Start) User has no preferences in Redis or Bloom filter. What are defaults? Default preferences stored per-topic in config (not per-user). If no user preference found, apply topic default. Most topics default to "subscribed". Config-based defaults
Preference Write Storm Marketing campaign causes 1M users to unsubscribe simultaneously. Write buffer: batch preference updates into 1s windows. Async propagation to Redis. Bloom filter rebuild triggered on high-write-rate. <10s eventual propagation

Interview Cheat Sheet

5 key points for notification preference system design interviews

1. Bloom Filter for Fast Rejection is the Key Optimization
25B preference records can't all fit in memory. But a Bloom filter encoding 2B unsubscribe entries fits in 2.5GB. It rejects 70% of notifications in <0.05ms (in-process memory, no network). Only 30% need a Redis round-trip. This is the layered checking pattern: cheapest check first, expensive check only when needed.
2. Sliding Window Frequency Caps (Not Fixed Window)
Fixed window counters have the edge-of-window problem: 5 notifications at 11:59pm + 5 at 12:01am = 10 in 2 minutes. Sliding window (Redis Sorted Set with timestamp scores) ensures "max 5 in any rolling 24h period". ZCOUNT from (now - 24h) to now gives exact count. Atomic Lua script prevents race conditions.
3. Timezone by Name, Not Offset
Store "America/New_York" not "-05:00". The offset changes twice a year (EST→EDT). If you store offset, quiet hours break during DST transitions. IANA timezone names automatically handle DST via standard libraries. Calculate the current offset at check time — never cache it statically.
4. GDPR Compliance = Global Flag Checked First
When a user clicks "unsubscribe all", set a global flag in Redis. The preference check pipeline checks this flag BEFORE anything else. If set, suppress all non-essential notifications immediately. Propagation via pub/sub to all workers in <1s. Individual preference cleanup happens async — the flag provides instant protection.
5. Fail-Open vs Fail-Closed Depends on Notification Type
If Redis is down, what do you do? For security/transactional (OTP, password reset): fail-OPEN — send anyway, because not delivering is worse than violating a preference. For marketing: fail-CLOSED — suppress, because sending unwanted marketing to an unsubscribed user is a compliance violation. This nuanced approach shows system thinking.