System Design Case Study

How does a notification system deliver push notifications to millions of devices reliably?

🎯 Design a push notification system that delivers to millions of devices without duplicates, handling token lifecycle and cross-platform delivery
Concepts Involved
Message Queues Dead Letter Queue Idempotency Rate Limiting NoSQL

How does a notification system deliver push notifications to millions of devices without duplicates, handling device token lifecycle, retry logic, and cross-platform delivery (iOS/Android) reliably?

Scope: Push notification delivery to mobile devices when the app is backgrounded or the device is offline. The hard question: how do you guarantee exactly-once delivery semantics across unreliable APNs/FCM gateways while managing millions of device tokens that constantly change✗
1B+
push notifications / day
~12K/sec average
0
duplicate notifications
idempotency key per push
2
platforms (iOS + Android)
APNs + FCM integration
<2s
delivery P95
trigger → device display

Functional Requirements

What the system must do · reliable cross-platform push delivery with deduplication

Must Have (Core)

1. Deliver push notifications to iOS (APNs) and Android (FCM) devices
2. No duplicate notifications · idempotency key ensures exactly-once display
3. Handle device token lifecycle · registration, refresh, invalidation
4. Support priority levels · high (new message), normal (status update), low (marketing)
5. Badge count management · accurate unread count on app icon
6. Retry with backoff for transient APNs/FCM failures (not for invalid tokens)

Out of Scope (this design)

✗ In-app notifications (delivered via WebSocket · see Real-Time Messaging)
✗ Email/SMS notifications (different delivery channels, same trigger system)
✗ Notification preferences UI (user settings, mute, DND schedules)
✗ Rich media in notifications (images, action buttons · platform-specific)
✗ Analytics (open rates, click-through · separate pipeline)

Non-Functional Requirements

Quality constraints for reliable cross-platform push delivery at billion-device scale

PropertyTargetWhy It Matters / Design Impact
Delivery latencyP95 <2s from trigger to device displayNotifications must arrive while context is fresh · delayed push loses relevance.
Throughput1B+ notifications/day (~12K/sec avg, 50K/sec peak)Must sustain high throughput without backpressure on message delivery pipeline.
DeduplicationZero duplicate notifications (idempotency key)Duplicate buzzes annoy users and erode trust. Idempotency key per push event.
ReliabilityAt-least-once trigger, at-most-once displayNever miss a trigger event, but never show the same notification twice on device.
Token freshnessHandle 10M+ token rotations/dayStale tokens cause silent delivery failures. Must detect and refresh proactively.
Platform coverageiOS (APNs) + Android (FCM) + Web (WebPush)Each platform has different protocols, rate limits, and error semantics.
BatchingCollapse multiple msgs in same chat into single notificationPrevents notification spam when user receives burst of messages in one conversation.

Scale Estimation

Deriving push volume from message patterns and device registry size

StepCalculationResult
12B users · 50% with push enabled1B push-eligible devices
2100B msgs/day · 30% to offline users30B offline msgs → triggers
3Collapse/batch: 30B triggers · 30 (avg batch)~1B actual pushes/day
41B · 86400~12K pushes/sec avg, peak 50K/sec
5Device registry: 1B devices · 200 bytes~200GB (Cassandra)
6Dedup store: 1B keys/day · 32 bytes · 24h TTL~32GB Redis

Architecture Overview

Event trigger → Notification Service → Platform Router → APNs/FCM delivery with retry and dedup

TRIGGER PATH · Message arrives for offline user → push notification Message Event user is offline (from offline delivery) Notification Service dedup check (idempotency) priority assignment + batching Device Token Registry user_id → [{token, platform}] handles multi-device APNs Sender HTTP/2 persistent conn Apple APNs FCM Sender HTTP/2 batch API Google FCM ➔ iOS ➔ Android DEVICE TOKEN LIFECYCLE · Registration, Refresh, Invalidation Registration App launch → get token from OS POST /v1/devices {token, platform, user_id} Upsert: same device_id updates token Token Refresh OS rotates token periodically App detects change → re-register Old token invalidated automatically Invalidation APNs/FCM returns "invalid token" Mark token as dead in registry Causes: app uninstall, token rotation Deduplication idempotency_key = hash(user_id + msg_id) Redis SET with 24h TTL Prevents retry storms from causing dupes RETRY LOGIC & PRIORITY LEVELS Retry Strategy Transient failure (5xx, timeout): retry 3· with exponential backoff Invalid token (410 Gone): delete token, no retry Rate limited (429): backoff per APNs/FCM guidance Priority Levels HIGH: new message → immediate delivery, wake device NORMAL: status update → batched by OS LOW: marketing → silent, OS may delay hours Batching for Efficiency Group chat: 5 msgs in 10s → 1 push "5 new messages" Collapse key: same chat → replace previous push Badge count: atomic increment per notification

Key Design Decisions

Reliability and deduplication are the primary concerns · not speed

DecisionChoiceWhyAlternative Rejected
DeduplicationIdempotency key per notificationhash(user_id + message_id) stored in Redis with 24h TTL. Prevents duplicate pushes on retry.No dedup: users receive same notification 2-3· on transient failures. Unacceptable UX.
Token StorageDevice Registry (Cassandra)user_id → list of {device_id, token, platform, last_seen}. Supports multi-device. Handles token rotation.Single token per user: breaks multi-device. In-memory: lost on restart.
Platform RoutingSeparate sender per platformAPNs and FCM have different APIs, rate limits, and error codes. Separate senders allow independent scaling.Unified sender: complex error handling, can't scale platforms independently.
Trigger MechanismKafka consumer on offline-delivery topicSame event that triggers offline message storage also triggers push. Decoupled from message delivery path.Synchronous push on message send: adds latency to sender, couples systems.
Collapse StrategyCollapse key = conversation_idMultiple messages in same chat → single notification updated. Reduces notification fatigue.One push per message: 50 messages = 50 notifications. Terrible UX in active group chats.
Cross-reference: The push notification system is triggered by the same offline delivery mechanism described in WhatsApp Offline Delivery. When a message can't be delivered via WebSocket (user offline), it's queued for offline delivery AND triggers a push notification.

Resilience & Edge Cases

Push delivery is inherently unreliable · the system must handle every failure mode gracefully

#Failure / Edge CaseWhat BreaksHow It's Handled
1APNs returns "invalid token" (410)Token is stale · app was uninstalled or token rotatedMark token as dead in registry. Do not retry. Wait for app to re-register with new token on next launch.
2FCM rate limit (429)Too many requests to FCM · pushes delayedExponential backoff per FCM's Retry-After header. Queue notifications in Kafka · no data loss.
3User has 3 devices, message sent to allUser reads on one device, still gets push on othersOn message read (any device), cancel pending pushes for that message via collapse key update.
4Notification service crashes mid-batchSome notifications sent, some not · risk of duplicates on restartIdempotency key check before every send. On restart, re-process Kafka offset · dupes are caught by Redis dedup set.
5Badge count driftApp icon shows wrong unread countBadge count is server-authoritative. On each push, include absolute badge count (not increment). App syncs on foreground.
Key insight: Push notifications are best-effort with dedup. You can't guarantee delivery (device may be off, in airplane mode, or have notifications disabled). But you CAN guarantee no duplicates · which is what users actually notice and complain about.

Tech Stack & Tradeoffs

Optimized for reliability and deduplication over raw speed

ComponentTechnologyWhy ThisWhy Not X
Event SourceKafka (offline-delivery topic)Same event triggers offline storage + push. Decoupled, replayable. Consumer group for push senders.Direct call from message service: couples systems, no replay on failure.
Device RegistryCassandraHigh write throughput for token updates. Partition by user_id. Multi-device support. No single point of failure.MySQL: write bottleneck at scale. Redis: not durable enough for token registry.
Dedup StoreRedis (SET with TTL)O(1) idempotency check. 24h TTL auto-cleans. Handles retry storms without duplicates.Cassandra: too slow for per-notification check. In-memory: lost on restart.
APNs ClientHTTP/2 persistent connectionApple requires HTTP/2. Persistent connection avoids TLS handshake per push. Multiplexed streams.Legacy binary protocol: deprecated. New connection per push: 100ms+ TLS overhead.
FCM ClientHTTP/2 + batch APISend up to 500 notifications per batch request. Reduces HTTP overhead by 100·.Individual sends: 1 HTTP request per notification · doesn't scale to millions/day.
Cross-reference: The trigger mechanism connects to the WhatsApp Offline Delivery system. When a user comes back online, they receive messages via the offline queue AND the push notification is automatically superseded (collapse key).

Interview Cheat Sheet

The 5 things an interviewer wants to hear about push notifications at scale

? Idempotency Key for Zero Duplicates
Every notification gets a unique key: hash(user_id + message_id). Before sending, check Redis SET. If key exists, skip. This handles retry storms, consumer restarts, and network timeouts without ever showing a duplicate notification.
? Device Token Lifecycle Management
Tokens are ephemeral · they change on app reinstall, OS update, or periodic rotation. Registry stores user_id → [{device_id, token, platform}]. On "invalid token" response from APNs/FCM, immediately mark dead. On app launch, always re-register.
? Collapse Key for Notification Batching
Set collapse_key = conversation_id. Multiple messages in same chat within a short window → single notification that updates (not stacks). "5 new messages from #general" instead of 5 separate notifications. Reduces notification fatigue dramatically.
? Separate Senders per Platform (APNs/FCM)
APNs and FCM have different APIs, rate limits, error semantics, and scaling characteristics. Separate sender services allow independent scaling, isolated failures, and platform-specific optimizations (HTTP/2 multiplexing for APNs, batch API for FCM).
? Retry Only Transient Failures
Critical distinction: 5xx/timeout → retry with exponential backoff (transient). 410 Gone → delete token, never retry (permanent). 429 → backoff per Retry-After header. Wrong retry strategy = wasted resources or lost notifications.
One-liner answer: "Kafka-triggered push with idempotency key dedup in Redis, device token registry in Cassandra with lifecycle management, collapse keys for notification batching per conversation, separate APNs/FCM senders with platform-specific scaling, and retry only on transient failures with exponential backoff."