How does a chat system handle millions of ephemeral typing events per second?
How does a chat system handle millions of ephemeral 'typing...' events per second without persisting anything to disk, while ensuring sub-100ms delivery to all conversation participants?
What the system must do · ephemeral event delivery with aggressive throttling
Quality constraints for ephemeral event delivery at massive scale
| Property | Target | Why It Matters / Design Impact |
|---|---|---|
| Latency | P99 <100ms delivery | Typing must feel instant · any perceptible delay breaks the illusion of real-time presence. |
| Throughput | 5M+ events/sec peak | All active users typing simultaneously during peak hours. Must handle without queuing. |
| Durability | Zero (ephemeral, never persisted to disk) | Typing indicators are transient signals · persisting them wastes I/O and adds no value. |
| Availability | 99.9% (best-effort) | Brief outage is invisible · users won't notice a missed typing indicator. |
| Delivery | At-most-once | Missed indicator is unnoticeable. Duplicate would cause UI flicker · worse than missing one. |
| Bandwidth | Minimal per event (~50 bytes payload) | High event rate means even small payloads add up. Keep wire format tiny. |
Deriving throughput requirements from concurrent user behavior
| Step | Calculation | Result |
|---|---|---|
| 1 | 10M concurrent users · 20% actively typing | 2M typing users |
| 2 | 2M users · 1 event/5s (throttled) | 400K events/sec avg |
| 3 | Peak factor 10· | ~4-5M events/sec |
| 4 | Avg conversation size 5 people → 400K · 4 recipients | 1.6M deliveries/sec |
| 5 | Redis Pub/Sub capacity: ~1M msgs/sec per node | Need 2-3 Redis nodes |
| 6 | Memory for throttle state: 2M users · 32 bytes | ~64MB per gateway (trivial) |
Direct gateway-to-gateway pub/sub · no Kafka, no database, pure in-memory routing for sub-100ms delivery
Every decision optimizes for one thing: minimum latency with zero durability
| Decision | Choice | Why | Alternative Rejected |
|---|---|---|---|
| Transport | Redis Pub/Sub (gateway-to-gateway) | Sub-ms fan-out, no persistence, fire-and-forget semantics match typing events perfectly | Kafka: adds 50ms+ latency, persists to disk (wasteful for ephemeral data) |
| Throttling | Dual: client + server | Client reduces 60 keystrokes/min to 1 event/5s. Server enforces for misbehaving clients. | Server-only: wastes bandwidth sending events that will be dropped |
| Expiry | TTL-based (8s client timer) | No "stopped typing" event needed · reduces event volume by 50%. Timer resets on new event. | Explicit stop event: doubles event volume, complex state machine |
| Delivery guarantee | At-most-once (fire & forget) | Missed typing indicator is invisible to user · no retry needed. Simplifies entire system. | At-least-once: requires ACKs, retries, dedup · overkill for ephemeral UI hint |
| Routing | Channel-based pub/sub topic | Subscribe to typing:{channel_id}. Only gateways with online members subscribe. | User-level routing: too many subscriptions, O(members) lookups per event |
Typing indicators are lossy by design · the system optimizes for graceful degradation over correctness
| # | Failure / Edge Case | What Breaks | How It's Handled |
|---|---|---|---|
| 1 | Gateway crash | Typing events for users on that gateway are lost | Acceptable · typing indicator disappears after 8s TTL. No recovery needed. Users reconnect to new gateway. |
| 2 | Redis Pub/Sub node failure | Events not delivered to subscribed gateways | Redis Sentinel promotes replica in ~5s. During failover, typing indicators simply don't show · invisible to users. |
| 3 | User types then closes app | "Typing..." shown for up to 8s after user left | 8s TTL auto-expires. Presence system detects disconnect and can optionally clear early. |
| 4 | 50-person group chat, all typing | 50 typing events · 49 recipients = 2,450 deliveries | Client UI caps display at "3 people are typing..." · server can further throttle to top-3 most recent. |
| 5 | Misbehaving client sends 100 events/sec | Could flood pub/sub channel | Server-side throttle drops events within 5s window. Rate limit at Gateway: max 1 event/5s enforced. |
Minimal stack · optimized for speed and simplicity over durability
| Component | Technology | Why This | Why Not X |
|---|---|---|---|
| Event Transport | Redis Pub/Sub | Sub-ms delivery, fire-and-forget, no persistence overhead. Channel-based subscription model. | Kafka: persists to disk, adds 50ms+ latency. Overkill for ephemeral events. |
| Throttle State | In-memory HashMap (per Gateway) | O(1) lookup, no network hop. Map: {user+channel → last_event_ts}. Evicted on disconnect. | Redis: adds 1ms per event for throttle check · unnecessary network hop. |
| Client Timer | setTimeout (8s TTL) | Auto-expires indicator without server involvement. Reset on each new event. | Server-sent "stop" event: doubles traffic, complex state management. |
| WebSocket Gateway | Same as message delivery (Go/Java) | Reuses existing WS connections. Typing events are just another frame type on the same socket. | Separate connection: wasteful, doubles connection count. |
| Subscription Management | Redis Pub/Sub channels per conversation | Gateway subscribes to typing:{channel_id} when first user in that channel connects. | Per-user subscriptions: O(members) subscriptions per channel, doesn't scale. |
The 5 things an interviewer wants to hear about typing indicators at scale
typing:{channel_id} for channels with connected users. When a typing event arrives, the Gateway pushes to all local users in that channel. Unsubscribe when last user in channel disconnects.