Notification Service Architecture

🎯 Design a notification service that delivers push/email/SMS/in-app notifications to 500M users with priority queuing, channel routing, delivery guarantees, template rendering, and preference checking

Concepts Involved

Priority Queue Fan-out At-Least-Once Delivery Channel Routing Worker Pools

Problem Statement

How do you build a notification service that fans out to 4 channels (push/email/SMS/in-app), manages device tokens for 500M users, guarantees delivery, renders templates, and checks preferences — all within 10s P99?

Scope: Core notification service — priority queue ingestion, channel routing based on user preferences, per-channel delivery workers, device token management, template rendering engine. Not covering preference management UI or analytics dashboards.

500M

registered users

multi-device, multi-channel

5B/day

notifications sent

~58K notifications/sec avg

delivery channels

push / email / SMS / in-app

<10s

delivery P99 latency

end-to-end from trigger

Functional Requirements

What the notification service must do · core delivery behaviours

Must Have (Core)

1. Accept notification events via API and route to priority queue
2. Channel routing · determine which channels (push/email/SMS/in-app) based on user preferences
3. Template rendering · resolve notification templates with dynamic params per channel
4. Device token management · store/refresh APNs, FCM tokens per user per device
5. Per-channel workers · dedicated worker pools for push, email, SMS, in-app delivery
6. At-least-once delivery · retry failed deliveries with exponential backoff

Out of Scope (this design)

✗ User preference management UI (separate microservice)
✗ Analytics and reporting (open rates, click-through)
✗ A/B testing of notification content
✗ Rich media rendering (images, deep links in push)
✗ Billing/cost management for SMS/email providers
✗ Compliance (GDPR opt-out, CAN-SPAM) enforcement logic

Non-Functional Requirements

Quality constraints shaping the notification service architecture

Property	Target	Why It Matters / Design Impact
Delivery Guarantee	At-least-once	Retry with backoff on provider failure. Idempotency keys prevent duplicate sends from user perspective.
Latency (P99)	<10s end-to-end	Priority queue ensures critical notifications (OTP, alerts) bypass marketing batch. Pre-rendered templates cached.
Availability	99.99%	Multi-AZ deployment. Queue durability ensures no notification lost even during deployments.
Throughput	58K+ notifications/sec	Horizontally scaled workers per channel. Kafka partitions for parallel consumption.
Multi-Channel	4 channels simultaneously	Each channel has independent worker pool, retry policy, and provider failover.
Scalability	Linear with user growth	Stateless workers. Partition by user_id for token lookups. Add workers per channel independently.
Observability	Per-notification tracking	Delivery log for every notification: sent, delivered, failed, opened. SLA alerting on delivery rate drops.

Key tension: Latency vs. Throughput. High-priority notifications (OTP, security alerts) need <3s delivery, while marketing batches can tolerate minutes. Solved with a multi-priority queue where P0 notifications always dequeue first.

Scale Estimation

Derive infrastructure sizing from given constraints. Show this reasoning in an interview.

Given: 500M users · 5B notifications/day · 4 channels · <10s P99 · avg 2 devices/user

Step	What to Derive	Calculation	Result	Design Decision
1	Avg notifications/sec	5B / 86,400s	~58K/sec	Kafka with 128 partitions handles this with headroom for burst
2	Peak notifications/sec	58K × 5 (burst during events)	~290K/sec peak	Auto-scaling worker pools. Queue absorbs burst, workers drain at capacity.
3	Device tokens stored	500M users × 2 devices avg	1B device tokens	Sharded Redis for hot lookups + DynamoDB for persistence
4	Push worker pool size	60% of 58K = 35K push/sec · 500 push/sec/worker	~70 push workers	Batch API calls to APNs/FCM (500 tokens per batch request)
5	Email worker pool size	25% of 58K = 14.5K emails/sec · 1000 emails/sec/worker	~15 email workers	SendGrid/SES batch API. Template pre-rendered, just inject params.
6	SMS worker pool size	5% of 58K = 2.9K SMS/sec · 200 SMS/sec/worker	~15 SMS workers	Twilio rate limits per number. Pool of sender numbers for throughput.
7	Delivery log storage/day	5B logs × 200 bytes each	~1TB/day	Time-partitioned table in Cassandra. TTL 30 days for compliance.
8	Redis memory (tokens + prefs)	1B tokens × 128 bytes + 500M prefs × 64 bytes	~160GB	Redis cluster (32 shards, 5GB each). LRU eviction for cold users.

Interview tip: Start with throughput (step 1-2), then show you understand per-channel capacity planning (steps 4-6). The key insight is that each channel has wildly different throughput characteristics — push is batch-friendly, SMS is rate-limited per number.

APIs

Core notification service endpoints · RESTful with priority support

POST /v1/notifications/send

Enqueue a notification for delivery across configured channels

{
  "user_id": "u_abc123",
  "template_id": "order_shipped",
  "priority": "high",           // high | medium | low
  "channels": ["push", "email"],
  "params": {
    "order_id": "ORD-9876",
    "tracking_url": "https://track.example.com/9876"
  },
  "dedup_key": "order_shipped:ORD-9876:u_abc123",
  "schedule_at": null           // null = immediate, ISO8601 = delayed
}

Response: 202 Accepted · { "notification_id": "n_xyz789", "status": "queued" }

GET /v1/notifications/{user_id}/feed

Retrieve in-app notification feed with pagination

GET /v1/notifications/u_abc123/feed?limit=20&cursor=eyJ0IjoiMjAyNC0wMS0xNSJ9

Response 200:
{
  "notifications": [
    { "id": "n_xyz789", "title": "Order Shipped", "body": "...", "read": false, "created_at": "..." }
  ],
  "next_cursor": "eyJ0IjoiMjAyNC0wMS0xNCJ9",
  "unread_count": 7
}

PUT /v1/notifications/{id}/read

Mark a notification as read (updates in-app feed)

PUT /v1/notifications/n_xyz789/read

Response 200: { "id": "n_xyz789", "read": true, "read_at": "2024-01-15T10:30:00Z" }

Data Model

Core tables and Redis structures powering the notification pipeline

notification_events (Cassandra)

CREATE TABLE notification_events (
  notification_id   UUID PRIMARY KEY,
  user_id           TEXT,
  template_id       TEXT,
  priority          TEXT,          -- high | medium | low
  channels          SET<TEXT>,     -- {push, email, sms, inapp}
  params            MAP<TEXT,TEXT>,
  dedup_key         TEXT,
  status            TEXT,          -- queued | processing | delivered | failed
  created_at        TIMESTAMP,
  delivered_at      TIMESTAMP
);

user_devices (DynamoDB)

{
  "user_id":      "u_abc123",          // partition key
  "device_id":    "dev_001",           // sort key
  "platform":     "ios",               // ios | android | web
  "push_token":   "APNs:abc...",       // provider-specific token
  "token_updated": "2024-01-10T...",
  "app_version":  "4.2.1",
  "active":       true
}

delivery_log (Cassandra, time-partitioned)

CREATE TABLE delivery_log (
  notification_id   UUID,
  channel           TEXT,          -- push | email | sms | inapp
  attempt           INT,
  status            TEXT,          -- sent | delivered | failed | bounced
  provider          TEXT,          -- apns | fcm | sendgrid | twilio
  provider_msg_id   TEXT,
  error_code        TEXT,
  attempted_at      TIMESTAMP,
  PRIMARY KEY ((notification_id), channel, attempt)
) WITH default_time_to_live = 2592000;  -- 30 day TTL

Redis Structures

// Device tokens for fast lookup during send
HSET device_tokens:{user_id} {device_id} "{platform}:{push_token}"

// User preferences cache (channel subscriptions)
HSET preferences:{user_id} "push" "true" "email" "true" "sms" "false"

// In-app notification feed (sorted set by timestamp)
ZADD inapp_feed:{user_id} {timestamp} {notification_id}

// Unread counter
INCR unread:{user_id}

Architecture Overview

Event Source → Priority Queue → Channel Router → Per-Channel Workers → Delivery Providers

Key Design Decisions

Tradeoffs made in this architecture and their rationale

Decision	Options Considered	Chosen	Why
Queue Technology	RabbitMQ, SQS, Kafka	Kafka	Durable, replayable, high throughput. Priority via separate topics. Consumer groups for scaling workers.
Priority Implementation	Single queue with priority field, separate queues per priority	Separate topics per priority	P0 consumers never blocked by P3 backlog. Independent scaling per priority level.
Template Storage	DB per render, pre-compiled cache	Pre-compiled + cached in Redis	Template rendering is CPU-bound. Pre-compile Mustache/Handlebars templates, cache in Redis. Invalidate on template update.
Device Token Store	PostgreSQL, DynamoDB, Redis-only	DynamoDB + Redis cache	DynamoDB for durability (tokens are critical). Redis for sub-ms lookup during send path. Write-through cache.
Delivery Guarantee	At-most-once, at-least-once, exactly-once	At-least-once + dedup	Provider ACKs are unreliable. Retry on failure. Dedup key prevents user-visible duplicates.
Worker Isolation	Shared workers, per-channel workers	Per-channel worker pools	SMS rate limits shouldn't block push delivery. Independent scaling, retry policies, circuit breakers per channel.

Why priority queues matter: An OTP code that arrives 30s late is useless. By separating P0 (critical) from P3 (marketing), we guarantee that a batch of 10M marketing emails never delays a single OTP delivery. P0 has dedicated consumers that are never backlogged.

Template rendering tradeoff: Rendering at send-time allows personalization but adds latency. Solution: pre-compile templates to bytecode, cache in Redis. At send-time, only variable injection happens (~0.1ms per notification). Template updates invalidate cache via pub/sub.

Provider failover: If APNs returns 503, don't retry immediately to APNs. Instead, circuit-break that provider and route push notifications to FCM (for Android) or fall back to in-app. Provider health is tracked per-region with sliding window failure rates.

Device token hygiene: APNs returns HTTP 410 (Gone) for uninstalled apps. FCM returns NotRegistered. Workers MUST delete stale tokens immediately. Sending to stale tokens wastes quota and can trigger provider throttling. Run daily sweep for tokens not updated in 30 days.

Resilience & Edge Cases

How the notification service handles failures gracefully

Scenario	Problem	Solution	Recovery
Provider Outage	APNs or Twilio returns 5xx for extended period.	Circuit breaker opens after 50% failure rate in 60s window. Messages queue up. Auto-retry when circuit closes. Alert on-call.	Auto-recover on provider restoration
Token Expiry Storm	App update causes mass token refresh — 100M tokens invalid simultaneously.	Batch token refresh via FCM/APNs canonical ID response. Queue token updates. Don't block send path — mark for async refresh.	~1 hour full refresh
Notification Flood	Upstream bug sends 100x normal volume.	Per-user rate limiting (max 10 notifications/min per user). Per-source rate limiting. Queue depth alerting + auto-scale workers.	Immediate suppression
Duplicate Events	Producer retries cause same notification enqueued twice.	Dedup key (user_id + template_id + params_hash) checked in Redis before processing. 24h TTL on dedup entries.	Zero user-visible duplicates
Template Corruption	Bad template deploy renders broken notifications.	Template versioning. Canary deploy to 1% traffic. Rollback on error rate spike. Fallback to previous version automatically.	<5 min auto-rollback
Queue Lag Spike	Consumer lag exceeds SLA threshold (P0 messages waiting >3s).	Auto-scale consumers on lag metric. Priority isolation ensures P0 consumers are never starved. Alert if lag persists after scale-up.	<30s auto-scale response

Interview Cheat Sheet

5 key points to nail a notification service architecture interview

1. Priority Queue is the Core Differentiator
Not all notifications are equal. OTP codes need <3s, marketing can wait minutes. Use separate Kafka topics per priority level with dedicated consumer groups. P0 consumers are over-provisioned and never share resources with lower priorities. This guarantees SLA per notification type.

2. Per-Channel Workers with Independent Scaling
Push, email, SMS, and in-app have wildly different throughput profiles. Push batches 500 tokens/request. SMS is rate-limited per number. Email is CPU-bound for template rendering. Isolating workers per channel prevents one slow channel from blocking others and enables independent auto-scaling.

3. Preference Check BEFORE Fan-out
The channel router checks user preferences before splitting to per-channel queues. This prevents wasted work — if a user opted out of SMS, we never enqueue to the SMS worker. Preferences are cached in Redis for <1ms lookup. Real-time opt-out propagation via pub/sub.

4. Device Token Management is Critical Infrastructure
1B device tokens across 500M users. Tokens expire, rotate, get invalidated on app uninstall. You must handle APNs 410 (Gone), FCM NotRegistered responses by immediately deleting stale tokens. A daily sweep removes tokens not refreshed in 30 days. Stale tokens waste provider quota and trigger throttling.

5. At-Least-Once + Dedup = User-Perceived Exactly-Once
Provider ACKs are unreliable (network timeout ≠ delivery failure). So we retry on any ambiguous failure. To prevent user-visible duplicates, every notification carries a dedup_key (user_id + template_id + params_hash). Workers check Redis SETNX before sending. 24h TTL on dedup entries covers retry windows.

System Design Case Study