System Design Case Study

How does WhatsApp guarantee message delivery to 2B+ users who go offline · with end-to-end encryption?

?? Design a messaging system that guarantees zero message loss for 2B+ users who go offline for hours/days, with E2E encryption where the server never sees plaintext
Concepts Involved
Message Queues Encryption (E2E) NoSQL Ordering & Consistency Sharding Multi-Region

How does a messaging system guarantee delivery to 2B+ users who go offline for hours/days, ensuring zero message loss, correct ordering on reconnect, and end-to-end encryption without the server ever seeing plaintext?

Key difference from Slack: Slack assumes users are mostly online (WebSocket push). WhatsApp must handle the offline-first case · users may be offline for days, on unreliable mobile networks, with intermittent connectivity. See Slack Real-Time Messaging for the online delivery path.
2B+
monthly active users
100B+
messages / day
E2E
encrypted (Signal Protocol)
0
messages lost after ACK

Functional Requirements

What the system must do · focused on offline delivery and encryption

Must Have (Core)

1. Messages never lost after sender gets delivery ACK (??)
2. Offline users receive all missed messages on reconnect · in correct order
3. End-to-end encryption · server stores only ciphertext, never plaintext
4. Delivery receipts: single tick (sent) ? double tick (delivered) ? blue tick (read)
5. Messages expire from server after delivery (30-day TTL for undelivered)
6. Support media messages (photos, video, documents) with same guarantees

Out of Scope

? Group messaging fan-out (see Slack Fan-out Strategy)
? Real-time online delivery path (see Slack Architecture)
? Voice/video calling signaling
? Status/Stories feature
? Business API and chatbots

Non-Functional Requirements

Constraints shaped by mobile-first, offline-heavy usage patterns

PropertyTargetDesign Impact
DurabilityZero message loss after sender ACKWrite-ahead log + synchronous replication before ACK. Messages stored until recipient confirms delivery.
Latency<300ms when both onlineDirect push via persistent connection when recipient is online. No store-and-forward delay.
Offline toleranceUp to 30 days offlineServer queues messages per recipient. TTL-based expiry after 30 days. Recipient pulls on reconnect.
SecurityE2E encrypted · server is zero-knowledgeSignal Protocol (Double Ratchet). Server stores ciphertext only. Key exchange happens client-to-client.
OrderingPer-conversation ordering preservedSender-assigned sequence numbers. Recipient reorders on delivery. No global ordering needed.
BandwidthMinimal · 2G/3G friendlyBinary protocol (not JSON). Compressed payloads. Resumable media uploads. Delta sync on reconnect.
Scale2B+ users, 100B+ msg/dayShard by user_id. Each user's queue is independent. Horizontal scale with no cross-shard coordination.

Scale Estimation

Derive infrastructure sizing from the given numbers

StepDerivationResultDesign Decision
1100B msgs/day · 86,400s~1.15M msg/sec10· Slack's throughput · need sharded message queues
22B users · ~30% online at any time~600M concurrent connections~1.2M connection servers (500K conn each)
3Avg user offline 8-16 hrs/day ? ~50 msgs queued~100B queued msgs at peakNeed efficient per-user message queue storage
4Avg message 1KB (encrypted) · 100B queued~100TB queued storageDistributed KV store sharded by recipient_id
5Media: 10% of msgs have media, avg 500KB~5PB media/dayBlob storage with CDN. Encrypted client-side before upload.
6Reconnect storm: 100M users come online in 1 hour (morning)~28K reconnects/secQueue drain must handle burst. Stagger delivery over seconds.

Architecture Overview

Store-and-forward with E2E encryption · server is a dumb encrypted mailbox

WhatsApp Message Flow · Store-and-Forward with E2E Encryption Sender (Alice) Encrypts with Bob's public key (Signal) ciphertext Connection Server Holds Alice's TCP conn Validates, routes Message Queue (per-user) Stores ciphertext until recipient confirms delivery Bob ONLINE ? push immediately via Bob's TCP connection Bob OFFLINE ? store in queue + send push notification (FCM) Bob decrypts with private key ? ?? RECONNECT FLOW · Bob comes back online after 8 hours Bob connects Auth + send last_seen_seq Drain queue ? push all pending Bob ACKs each msg ? ?? Server deletes from queue Result: Bob receives all 50 missed messages in order. Server queue is now empty. Alice sees ?? for each. E2E ENCRYPTION · Signal Protocol (Double Ratchet) Key Exchange (X3DH) Identity key + signed prekey + one-time prekey ? shared secret Happens once per conversation Double Ratchet (per message) Each msg gets unique key Forward secrecy: compromise 1 key ? can't decrypt past/future msgs Server sees ONLY ciphertext Cannot decrypt even with court order Metadata visible: who ? whom, when Content: fully opaque to server Recipient decrypts locally Private key never leaves device Backup encryption: separate key managed by user (not WhatsApp)

Per-User Message Queue

Each user has an independent queue · no cross-user coordination needed. Sharded by recipient_id.

Per-User Queue · Append-Only Log with ACK-Based Deletion Queue for user:Bob (recipient_id = B456) msg_seq=1 msg_seq=2 msg_seq=3 msg_seq=4 seq=5 ? Delivered (ACKed) · will be deleted ? Pending delivery · Append-only: new messages added at tail · Bob's last_ack_seq = 3 ? server knows to deliver from seq=4 · After Bob ACKs seq=4,5 ? those entries deleted (or marked) · TTL: undelivered msgs expire after 30 days Storage: Mnesia / Custom KV (Erlang) Key: (recipient_id, msg_seq) Value: { sender_id, ciphertext, timestamp, media_url? } Shard key: recipient_id (all of Bob's msgs on one node) Replication: 3 replicas across AZs (sync write to 2) Deletion: On recipient ACK ? hard delete (privacy) Capacity: ~100TB total queued msgs across all users
Key difference from Slack: Slack stores messages permanently in Vitess (they're the system of record). WhatsApp deletes messages from the server after delivery · the server is just a temporary mailbox. The phone is the system of record. See Slack Data Model for the permanent storage approach.

Delivery Receipts (Tick System)

Three states: ? sent (server received) ? ?? delivered (recipient got it) ? ?? read (recipient opened)

Message Lifecycle · Receipt State Machine PENDING clock icon client ? server in-flight server ACK SENT ? single grey tick stored in recipient queue recipient ACK DELIVERED ?? double grey ticks msg reached device, decrypted read receipt READ ?? double blue ticks user opened conversation server cleanup DELETED removed from server only exists on devices Timing: PENDING?SENT: ~100ms | SENT?DELIVERED: instant (online) or hours/days (offline) | DELIVERED?DELETED: immediate after ACK

Receipt Protocol

Delivery ACK: Recipient sends ACK with msg_seq after decryption
Read receipt: Sent when user opens the chat (can be disabled)
Batch ACK: On reconnect, ACK all received msgs in one batch
Retry: If ACK lost, server re-delivers on next connection

Server Deletion Policy

After delivery ACK: Message deleted from server immediately
Media: Blob deleted from CDN after recipient downloads
TTL expiry: Undelivered msgs deleted after 30 days
Privacy: Server retains zero message content long-term

Failure Handling

ACK lost: Server re-delivers; client deduplicates by msg_seq
Device lost: Messages gone (unless backed up to encrypted cloud backup)
Server crash: Replicated queue survives; client reconnects to replica
Network flap: Client retries with exponential backoff + jitter

Ordering & Consistency

Per-conversation ordering via sender-assigned sequence numbers · no global ordering needed

How Ordering Works

1. Sender assigns msg_seq per conversation (monotonic on sender device)
2. Server appends to recipient queue preserving sender order
3. Recipient receives in queue order · reorders by timestamp if needed
4. Cross-conversation: no ordering guarantee (different queues)
5. Group messages: server assigns group_seq (single writer per group)

Difference from Slack

Slack: Server assigns channel_seq via Kafka partition (centralized)
WhatsApp: Sender assigns seq (decentralized) · works offline
Why: WhatsApp sender may be offline when server processes msg
Tradeoff: Slightly weaker ordering guarantee but works without server
See Slack Ordering for the centralized approach
Key insight: WhatsApp uses sender-assigned timestamps + per-conversation sequence because the sender might be on a flaky mobile connection. Slack uses server-assigned sequence because all senders are connected to the same Kafka partition. Different connectivity assumptions ? different ordering strategies.

Media Message Handling

Photos, videos, documents · encrypted client-side, uploaded to blob storage, link sent in message

Media Upload Flow · Client-Side Encryption Alice generates random AES-256 key Encrypt media locally AES-256-CBC + HMAC-SHA256 Upload ciphertext to blob storage (CDN) Send msg with {url, key, hash} key encrypted with Bob's Signal key Bob downloads + decrypts verify HMAC ? decrypt with key Server stores encrypted blob but CANNOT decrypt it (no key). Blob deleted after recipient downloads. Resumable upload: chunked with per-chunk checksum. Works on 2G/3G with intermittent connectivity.

Tech Stack & Tradeoffs

Erlang/BEAM for massive concurrency · each user connection is a lightweight process

ComponentTechnologyWhy ThisWhy Not X
Connection ServerErlang/BEAM2M+ connections per server. Lightweight processes (2KB each). Hot code reload. Built for telecom reliability.Go/Java: heavier per-connection overhead. Can't match Erlang's process density.
Message QueueMnesia (Erlang built-in)Distributed, replicated, in-memory with disk persistence. Co-located with connection server · no network hop.Kafka: overkill for per-user queues. Redis: no built-in replication at this scale.
Media StorageCustom blob store + CDNEncrypted blobs. Geo-distributed. Auto-delete after download. Resumable uploads.S3: vendor lock-in. Cost at 5PB/day would be enormous.
EncryptionSignal Protocol (libsignal)Forward secrecy. Post-compromise security. Well-audited. Open source.PGP: no forward secrecy. TLS: only transport-level, server can read.
Push NotificationsFCM / APNsWake device when offline. "You have a new message" (no content · E2E).Custom push: can't wake iOS apps without APNs. Platform requirement.
ProtocolCustom binary (XMPP-derived)Minimal bandwidth. Binary framing. Compression. 2G-friendly.JSON/HTTP: too verbose for mobile. gRPC: protobuf overhead unnecessary.
Real-world validation: WhatsApp famously handled 2B users with ~50 engineers and ~1000 servers using Erlang. The BEAM VM's actor model maps perfectly to "one process per user connection" · each process holds the user's state, mailbox, and connection.

Resilience & Edge Cases

Mobile-first challenges that don't exist in desktop-first systems like Slack

#ChallengeWhat BreaksHow It's Handled
1Phone lost/stolenAll messages on device goneEncrypted cloud backup (Google Drive / iCloud). Backup key managed by user, not WhatsApp. Restore on new device.
2SIM swap attackAttacker registers with victim's numberRe-registration triggers key change notification to contacts. 2FA PIN prevents unauthorized re-registration.
3Network flap (2G/3G)Connection drops mid-messageClient retries with exponential backoff. Server deduplicates by client_msg_id. Resumable media uploads.
4Morning reconnect storm100M users come online simultaneouslyStagger queue drain. Priority: recent msgs first. Rate-limit per-user delivery to 100 msgs/sec.
5Key mismatch (new device)Can't decrypt messages encrypted with old keySender re-encrypts pending messages with new key. "Security code changed" notification shown to contacts.
6Group key distributionN members need the same message, each with different keySender encrypts once per recipient (fan-out at encryption layer). Server stores N ciphertexts for N members.
Key insight: WhatsApp's architecture is fundamentally store-and-forward · the server is a temporary encrypted mailbox. This is the opposite of Slack's persistent log approach. The tradeoff: WhatsApp can't offer server-side search (content is encrypted), but provides stronger privacy guarantees.

Interview Cheat Sheet

The 5 things an interviewer wants to hear for this problem

? Store-and-forward with per-user queues
Each user has an independent message queue sharded by recipient_id. Messages stored as ciphertext until recipient ACKs delivery. Server deletes after ACK · it's a temporary mailbox, not permanent storage.
? E2E encryption via Signal Protocol
X3DH key exchange + Double Ratchet. Each message gets a unique key. Forward secrecy: compromising one key doesn't expose past/future messages. Server sees only ciphertext · can't decrypt even with a court order.
? Delivery receipts as state machine
PENDING ? SENT (?) ? DELIVERED (??) ? READ (blue ??) ? DELETED from server. Each transition triggered by an ACK from the next hop. Server cleanup happens immediately after delivery confirmation.
? Reconnect = drain queue from last_ack_seq
On reconnect, client sends last_ack_seq. Server delivers all messages with seq > last_ack_seq. Client ACKs each batch. Deduplication by msg_seq handles retries. Morning storm handled by staggered delivery.
? Erlang/BEAM for 2M connections per server
Each user connection is a lightweight Erlang process (2KB). Mnesia for co-located message queues. Hot code reload for zero-downtime deploys. 50 engineers, ~1000 servers for 2B users.
One-liner: "Per-user message queue sharded by recipient_id, E2E encrypted via Signal Protocol, store-and-forward with ACK-based deletion, delivery receipts as state machine, Erlang/BEAM for 2M connections per server."