Design outbound rate limiting for an SMS service that must not exceed a carrier's fixed acceptance rate
Design rate limiting for an outbound SMS service where the downstream carrier enforces a strict, fixed acceptance rate (100 messages/second). Bursts are not tolerated — the carrier will drop messages. The system must queue overflow, drain at a smooth constant rate, and handle backlog gracefully.
What the system must do · outbound rate limiting for a fixed-rate downstream carrier
Quality constraints shaping the leaky bucket design
| Property | Target | Why It Matters / Design Impact |
|---|---|---|
| Output Rate | EXACTLY 100 msg/sec | Carrier hard limit, no tolerance. Exceeding → messages silently dropped by carrier. |
| Queue Capacity | 10,000 messages | 100 seconds of buffer. Absorbs sustained bursts up to 25s at 500 msg/sec inbound. |
| Priority | OTP/2FA skip regular queue | Reserved 20% drain capacity (20 msg/sec) for critical messages. Never delayed by bulk. |
| Latency | Variable (depends on queue depth) | Max: capacity / drain_rate = 10,000 / 100 = 100 seconds worst case. |
| Overflow | 503 to upstream when queue full | Never silently drop. Upstream callers get explicit backpressure signal + Retry-After. |
Derive queue sizing and overflow timing from the given constraints
| Step | What to Derive | Calculation | Result | Design Decision |
|---|---|---|---|---|
| 1 | Queue fill rate during spike | 500 (inbound) - 100 (drain) = 400 msg/sec accumulation | 400 msg/sec net fill | Queue absorbs difference between inbound and outbound rates |
| 2 | Time to fill queue | 10,000 / 400 = 25 seconds of sustained peak | 25 seconds before overflow | Marketing campaigns must be time-boxed or rate-limited upstream |
| 3 | Memory for queue | 10K messages × 1KB avg size | ~10MB queue memory | Easily fits in Redis or in-memory buffer on single node |
| 4 | Priority lane capacity | 20% of 100 msg/sec drain reserved for OTP | 20 msg/sec OTP, 80 msg/sec regular | Critical messages always have guaranteed drain bandwidth |
| 5 | Time to drain full queue (no new input) | 10,000 / 100 = 100 seconds | 100s to fully drain | Max delivery latency during recovery from sustained overload |
External and internal interfaces for the SMS rate limiting service
Storage structures for queue, priority lanes, and overflow handling
// Queue: Redis LIST (RPUSH to enqueue, LPOP to drain)
// Or in-memory bounded buffer for single-node deployment
RPUSH sms:queue:regular {message_json} // Regular messages
RPUSH sms:queue:critical {message_json} // OTP/2FA messages
LPOP sms:queue:critical // Drain critical first
LPOP sms:queue:regular // Then drain regular
// Message structure
{
"id": "msg_abc123",
"to": "+1234567890",
"body": "Your OTP is 482910",
"priority": "critical",
"enqueued_at": 1705312200.123,
"ttl_expires_at": 1705312500.123
}
// DLQ: Separate Redis LIST for expired/failed messages
RPUSH sms:dlq {expired_message_json}
// Monitoring keys
GET sms:queue:regular:length // Current queue depth
GET sms:metrics:drain_rate // Actual drain rate (should be ~100/sec)
Messages enter a queue (bucket) and drain at a fixed, constant rate regardless of input pattern
| Decision | Choice | Why |
|---|---|---|
| Algorithm | Leaky bucket (FIFO queue + fixed drain rate) | Guarantees smooth, constant output. No burst ever reaches downstream. The textbook use case for leaky bucket. |
| Queue | Bounded FIFO queue (capacity: 10,000 messages) | Absorbs up to ~100s of burst. When full, reject with backpressure signal (HTTP 503 to upstream callers). |
| Drain | Timer-based: dequeue 1 message every 10ms (= 100/sec) | Tick-based drain ensures perfectly uniform output regardless of input pattern. |
| Overflow | When queue full: reject + DLQ for retry, or backpressure to upstream | Never drop silently. Upstream can retry or user sees delayed delivery. |
| Priority | Priority queue variant: OTP/2FA messages skip the regular queue | Critical messages (login OTP) must not be delayed by marketing bulk sends. |
| Monitoring | Queue depth + drain rate + overflow rate alerts | If queue stays > 80% for 5 minutes, scale capacity or shed low-priority load. |
| Failure | Impact | Recovery |
|---|---|---|
| Sustained overload (> 100 msg/sec for hours) | Queue fills up, delivery delays grow unbounded | Backpressure to upstream. Shed low-priority messages. Alert on queue depth > 80%. Consider additional carrier routes. |
| Carrier outage | Drain rate drops to 0, queue fills | Pause drain timer. Switch to backup carrier. Queue persists messages for retry when carrier recovers. |
| Message expires in queue | OTP delivered after timeout (useless) | TTL per message. Expired messages moved to DLQ. Application regenerates OTP if needed. |
| Priority inversion | Bulk marketing messages delay OTP | Separate priority lanes: critical (OTP) dequeues first. Critical lane gets 20% of drain capacity reserved. |
| Multiple carriers with different rates | Routing complexity | One leaky bucket per carrier. Router assigns messages to carrier with shortest queue. |
Key points for outbound rate limiting