Multi-Channel Notification Fallback

🎯 Design a notification system with cascading channel fallback: push → SMS → email, with configurable timeouts, state machine per notification, and guaranteed delivery via at least one channel

Concepts Involved

State Machine Delayed Queue Cascading Fallback Delivery Receipts Timeout Workers

Problem Statement

If push delivery fails (app uninstalled, token expired), fall back to SMS within 30s. Chain: push → wait 30s → SMS → wait 60s → email. Each channel has different delivery guarantees. How do you ensure every notification reaches the user via at least one channel?

Scope: Cascading fallback logic, per-notification state machine, delayed queue for timeout checks, delivery receipt processing, and deduplication across channels. Differs from core notification architecture by focusing on guaranteed delivery through channel escalation.

100M/day

notifications with fallback

critical + transactional only

15%

push failure rate

uninstalled / token expired

2 min

max end-to-end fallback

push(30s) + SMS(60s) + email

duplicate deliveries

once delivered, stop escalation

Functional Requirements

What the fallback system must do · guaranteed delivery via channel escalation

Must Have (Core)

1. Configurable fallback chain per notification type (push→SMS→email)
2. State machine per notification tracking current channel attempt and status
3. Timeout-based escalation · if no delivery receipt within window, try next channel
4. Delivery receipt processing · ingest provider webhooks (APNs, Twilio, SendGrid)
5. Immediate stop on delivery · cancel pending fallback timers once confirmed delivered
6. No duplicate sends · race condition between delivery receipt and timeout must be handled atomically

Out of Scope (this design)

✗ User preference management (handled by preference service)
✗ Template rendering (done before entering fallback pipeline)
✗ Cost optimization (SMS cheaper at certain hours)
✗ Channel-specific content adaptation (truncation for SMS)
✗ Analytics (fallback funnel analysis, channel success rates)
✗ Priority queue logic (upstream of this system)

Non-Functional Requirements

Quality constraints for cascading fallback delivery

Property	Target	Why It Matters / Design Impact
Guaranteed Delivery	99.9% via at least one channel	Fallback chain ensures if push fails, SMS catches it. If SMS fails, email is near-guaranteed. Must never silently drop.
Max Fallback Time	<2 minutes end-to-end	Push timeout 30s + SMS timeout 60s + email send = ~2 min worst case. User gets notification within 2 min regardless.
Zero Duplicates	0 duplicate deliveries	Atomic state transitions. If delivery receipt arrives while timeout fires, only one wins. Distributed lock on state transition.
State Consistency	Exactly one active attempt	At any moment, only one channel attempt is active per notification. State machine enforces this invariant.
Receipt Processing	<1s from provider webhook	Fast receipt processing cancels pending timers quickly, preventing unnecessary escalation to costlier channels.
Scalability	100M notifications/day	15M fallback to SMS daily (15% failure). State stored in Redis for sub-ms transitions.

Key tension: Speed vs. Cost. Shorter timeout catches failures faster but may trigger SMS before push actually delivers (provider delay ≠ delivery failure). Too long, and the user waits unnecessarily. 30s is the sweet spot — most push deliveries complete within 5s, so 30s gives adequate margin.

Scale Estimation

Derive fallback infrastructure sizing from constraints

Given: 100M notifications/day with fallback · 15% push failure · 30s push timeout · 60s SMS timeout · 5% SMS failure

Step	What to Derive	Calculation	Result	Design Decision
1	Notifications needing fallback/day	100M × 15% push failure	15M fall to SMS/day	SMS worker pool must handle 15M/day = ~174/sec additional load
2	SMS failures needing email	15M × 5% SMS failure	750K fall to email/day	Email is last resort — nearly 100% delivery rate via SendGrid/SES
3	Concurrent pending timers	~1,157 notif/sec × 30s window	~35K pending timers	Redis sorted set with score=expire_time. Timer worker polls every 1s.
4	State transitions/sec	1,157 new + 174 escalations + 1,157 receipts	~2,500 transitions/sec	Redis atomic operations (WATCH/MULTI or Lua script) for consistency
5	State storage (Redis)	35K active × 512 bytes per state object	~18MB active state	Fits easily in single Redis instance. Cluster for HA, not capacity.
6	Delivery receipt webhook volume	100M deliveries/day × 80% provide receipt	~926 webhooks/sec	Webhook ingestion service with Kafka buffer for spike absorption
7	Cost impact of fallback	15M SMS × $0.01/SMS + 750K emails × $0.001	$150K+/day SMS cost	Optimize push reliability first. Every 1% push improvement saves $10K/day.

Interview tip: Start with the failure rate (15% push failure is realistic), show the cascading math, then highlight the cost implications. The interviewer wants to see that you understand why minimizing fallback is critical — it's not just latency, it's $150K/day in SMS costs.

APIs

Endpoints for fallback-enabled notification delivery and status tracking

POST /v1/notifications/send

Send notification with fallback chain configuration

{
  "user_id": "u_abc123",
  "template_id": "otp_code",
  "params": { "code": "847291" },
  "fallback_chain": [
    { "channel": "push", "timeout_sec": 30 },
    { "channel": "sms", "timeout_sec": 60 },
    { "channel": "email", "timeout_sec": null }
  ],
  "dedup_key": "otp:u_abc123:847291",
  "max_ttl_sec": 300
}

Response: 202 Accepted · { "notification_id": "n_fb001", "state": "push_pending" }

GET /v1/notifications/{id}/status

Get current state and attempt history for a notification

GET /v1/notifications/n_fb001/status

Response 200:
{
  "notification_id": "n_fb001",
  "current_state": "delivered",
  "delivered_via": "sms",
  "attempts": [
    { "channel": "push", "status": "timeout", "started_at": "...", "ended_at": "..." },
    { "channel": "sms", "status": "delivered", "started_at": "...", "delivered_at": "..." }
  ],
  "total_time_ms": 34521
}

POST /v1/webhooks/delivery-receipt

Webhook endpoint for provider delivery confirmations

// APNs delivery receipt
{ "provider": "apns", "notification_id": "n_fb001", "status": "delivered", "timestamp": "..." }

// Twilio status callback
{ "provider": "twilio", "MessageSid": "SM...", "MessageStatus": "delivered", "To": "+1..." }

// SendGrid event
{ "provider": "sendgrid", "event": "delivered", "sg_message_id": "...", "timestamp": ... }

Data Model

State machine storage and delivery tracking structures

notification_state (Redis Hash)

HSET notification_state:{notification_id} 
  user_id           "u_abc123"
  current_state     "sms_pending"        // pending|push_sent|push_failed|sms_sent|sms_failed|email_sent|delivered|expired
  current_channel   "sms"
  attempt_index     1                    // which step in fallback_chain
  fallback_chain    '[{"channel":"push","timeout_sec":30},{"channel":"sms","timeout_sec":60},{"channel":"email"}]'
  created_at        "1705312200000"
  last_transition   "1705312230000"
  delivered_at      ""
  dedup_key         "otp:u_abc123:847291"

EXPIRE notification_state:{notification_id} 600   // TTL = max_ttl_sec + buffer

delayed_queue (Redis Sorted Set)

// Score = Unix timestamp when timeout fires
// Member = notification_id
ZADD delayed_queue 1705312230 "n_fb001"    // fires at created_at + 30s (push timeout)
ZADD delayed_queue 1705312290 "n_fb002"    // fires at created_at + 60s (sms timeout)

// Timer worker polls every second:
ZRANGEBYSCORE delayed_queue 0 {current_timestamp} LIMIT 0 100
// Process expired entries, then ZREM them

delivery_attempts (Cassandra — audit log)

CREATE TABLE delivery_attempts (
  notification_id   UUID,
  attempt_num       INT,
  channel           TEXT,          -- push | sms | email
  status            TEXT,          -- sent | delivered | failed | timeout
  provider          TEXT,          -- apns | fcm | twilio | sendgrid
  provider_msg_id   TEXT,
  error_code        TEXT,
  started_at        TIMESTAMP,
  completed_at      TIMESTAMP,
  PRIMARY KEY ((notification_id), attempt_num)
) WITH default_time_to_live = 2592000;  -- 30 days

State Machine Transitions

// Valid state transitions (enforced by Lua script):
pending       → push_sent      (push worker picks up)
push_sent     → delivered      (delivery receipt received)
push_sent     → push_timeout   (30s timer fires, no receipt)
push_timeout  → sms_sent       (sms worker picks up)
sms_sent      → delivered      (delivery receipt received)
sms_sent      → sms_timeout    (60s timer fires, no receipt)
sms_timeout   → email_sent     (email worker picks up)
email_sent    → delivered      (delivery receipt received)
*             → expired        (max_ttl exceeded)

Architecture Overview

Notification created → Push attempt → Timer → Check status → Escalate or confirm delivery

Key Design Decisions

Tradeoffs for cascading fallback architecture

Decision	Options Considered	Chosen	Why
State Storage	PostgreSQL, DynamoDB, Redis	Redis with Lua scripts	Atomic CAS required for race condition safety. Sub-ms transitions. TTL for auto-cleanup. State is ephemeral (max 5 min).
Timer Mechanism	OS timers, SQS delay, Redis ZSET	Redis Sorted Set	Score = expiry timestamp. Poll every 1s with ZRANGEBYSCORE. Survives worker restart. Easy to cancel (ZREM).
Timeout Duration	10s/30s/60s, 30s/60s/∞, adaptive	30s push / 60s SMS	95% of push deliveries complete in <5s. 30s gives margin for provider delays without excessive wait. SMS is slower (carrier delay).
Duplicate Prevention	Distributed lock, optimistic concurrency, CAS	Lua CAS (compare-and-swap)	Single Redis command, no external lock service. Atomic read-check-write in one operation. No deadlock risk.
Receipt Processing	Synchronous webhook, async queue	Kafka-buffered async	Provider webhooks can burst. Kafka absorbs spikes. Consumer processes receipts and updates state. Retry on failure.

Why not just retry the same channel? Push failure often means "app uninstalled" — retrying push 10 times won't help. The insight is that channel escalation (push→SMS→email) crosses different delivery infrastructure. SMS reaches the phone even when the app is gone. This is why fallback beats retry for notification delivery.

Timer precision matters: If your timer fires at 30.5s but the delivery receipt arrived at 30.1s, you still escalate unnecessarily. Solution: the timer doesn't directly send to the next channel. It first checks state via atomic CAS. If state already transitioned to "delivered", the timer is a no-op. This makes the system correct regardless of timing.

Cost explosion risk: At 15% push failure rate and 100M notifications/day, you're sending 15M SMS per day ($150K+). Monitor this rate. If push failure spikes to 30% (bad app update), SMS costs double. Set circuit breaker: if fallback rate exceeds 25%, alert on-call and optionally queue SMS instead of immediate send.

Idempotent escalation: The timer might fire twice (worker crash + restart). The Lua CAS script ensures only the first transition succeeds. Second attempt sees state ≠ expected, returns "already_transitioned", and the worker skips sending. This makes the entire pipeline idempotent despite at-least-once processing semantics.

Resilience & Edge Cases

Failure modes specific to cascading fallback delivery

Scenario	Problem	Solution	Recovery
Late Delivery Receipt	Push delivered at T=29s but receipt arrives at T=31s (after timeout). SMS already triggered.	CAS in Lua script: timer checks state before escalating. If push was delivered between timer fire and CAS, escalation is cancelled.	Zero duplicates via atomic CAS
Provider Webhook Down	Twilio can't reach our webhook — delivery receipt never arrives.	Don't rely solely on receipts. After max_ttl (5 min), mark as "presumed_delivered" if provider returned 200 on send. Reconcile async.	Eventual consistency via reconciliation
All Channels Fail	Push token expired, phone number invalid, email bounced.	After exhausting all channels in chain, mark as "undeliverable". Publish event for user health service. Flag user for contact info update.	Undeliverable → user flagged
Redis Failover	Redis primary fails, state lost during failover window.	Redis Sentinel with min-slaves-to-write=1. State also written to Kafka (event log). On recovery, replay Kafka events to rebuild state.	<5s failover + replay
Timer Worker Crash	Timer worker dies, pending timers not polled.	Multiple timer workers compete on ZRANGEBYSCORE with LIMIT + ZREM (atomic claim). Any worker can process any timer. No single-worker dependency.	<2s other worker picks up
Notification Expired	OTP code only valid for 5 min. Fallback chain takes 2 min. What if user reads at T=4 min?	Include expiry_at in notification content. Client-side: don't display expired notifications. Server-side: max_ttl cancels remaining chain on expiry.	Auto-expire at max_ttl

Interview Cheat Sheet

5 key points for notification fallback design interviews

1. State Machine is the Core Abstraction
Each notification has a finite state machine: pending → push_sent → (delivered | push_timeout → sms_sent → ...). This makes the system debuggable, auditable, and correct. At any point, you know exactly where a notification is in its lifecycle. State transitions are atomic via Redis Lua scripts.

2. The Race Condition is THE Hard Problem
Timer fires at T=30s. Delivery receipt arrives at T=30.1s. If both proceed, user gets duplicate notifications on different channels. Solution: Redis Lua compare-and-swap — both timer and receipt use the same CAS operation. Only one wins. Loser sees "already_transitioned" and becomes a no-op. This is the key insight interviewers are looking for.

3. Delayed Queue for Timeouts (Not OS Timers)
Don't use in-memory timers or setTimeout — they die with the process. Use Redis Sorted Set with score = expiry_timestamp. A timer worker polls ZRANGEBYSCORE every second. Advantages: survives crashes, multiple workers can compete, easy to cancel (ZREM), scales horizontally.

4. Cost Awareness: Fallback = Money
Each escalation step costs more: push ($0) → SMS ($0.01) → email ($0.001). At 15% push failure and 100M notifications/day, fallback costs $150K+/day in SMS. This means: (a) optimize push reliability aggressively, (b) set circuit breakers on fallback rate, (c) consider batching non-urgent fallbacks.

5. Not All Notifications Need Fallback
Only critical notifications (OTP, security alerts, payment confirmations) warrant the cost and complexity of cascading fallback. Marketing notifications should NOT escalate to SMS — that's a terrible user experience and expensive. Make fallback_chain configurable per notification type. Most notifications use single-channel delivery.

System Design Case Study