Design rate limiting for a payments API where clients run batch settlements (sudden bursts) but must average 100 req/sec
Design rate limiting for a payment API where merchants run batch settlements (e.g., 500 charges at once at end-of-day) as sudden bursts, but must average 100 requests/second over time. The system must deliberately allow controlled bursts while capping the sustained average rate.
What the system must do · burst-tolerant rate limiting for payment APIs
Quality constraints shaping the token bucket design
| Property | Target | Why It Matters / Design Impact |
|---|---|---|
| Burst | Allow up to 500 requests instantly | Batch settlement sends 500 charges at end-of-day. Must not be rejected. |
| Average | Sustained rate ≤ 100 req/sec | Long-term throughput cap. Prevents sustained abuse above contracted rate. |
| Recovery | Full burst restored after 5s idle | 500 tokens / 100 per sec refill = 5 seconds to full capacity. |
| Latency | <2ms per rate check | Redis atomic Lua script. Rate limit check must not add meaningful latency to payment flow. |
| Idempotency | Same key = no additional token cost | Payment retries are legitimate. Charging a token for retries would unfairly penalize reliable clients. |
Derive infrastructure sizing from merchant traffic patterns
| Step | What to Derive | Calculation | Result | Design Decision |
|---|---|---|---|---|
| 1 | Worst-case peak (all burst simultaneously) | 10K merchants × 500 burst = 5M requests | 5M (theoretical, unlikely) | Staggered in practice. Settlement times vary by timezone and merchant. |
| 2 | Realistic peak | 100 merchants settle at same time × 500 | 50K req/sec burst | Redis handles easily. Horizontal sharding of rate limit keys if needed. |
| 3 | Redis memory for rate limits | 10K keys × 2 fields (tokens, last_ts) × 50 bytes | ~1MB total | Trivial memory. Single Redis instance sufficient for rate limit state. |
| 4 | Refill calculation per request | tokens_added = elapsed_since_last × 100 (capped at 500) | O(1) per request | Lazy refill: no background timer. Two fields stored, one calculation on each check. |
| 5 | Idempotency cache size | 10K merchants × 100 req/sec × 24hr TTL × 200 bytes | ~17GB for full cache | Redis cluster with TTL eviction. Only store idempotency keys, not full responses. |
Payment API rate limiting interfaces and response headers
Storage structures for token bucket state and idempotency cache
// Redis HASH per merchant: token bucket state
// Key: ratelimit:merchant:{api_key}
HSET ratelimit:merchant:sk_live_abc123 tokens 350 last_refill 1705312200.123
// Lua script (atomic check + refill + deduct):
-- 1. Read current tokens and last_refill timestamp
-- 2. Calculate: elapsed = now - last_refill
-- 3. Add tokens: new_tokens = min(capacity, tokens + elapsed * refill_rate)
-- 4. If new_tokens >= 1: deduct 1, update timestamp, return ALLOWED
-- 5. Else: return DENIED + time_until_next_token
// Idempotency cache
// Key: idempotency:{api_key}:{idempotency_key}
SET idempotency:sk_live_abc123:idem_xyz "{cached_response}" EX 86400
// Check flow:
// 1. Check idempotency cache → if hit, return cached (no token cost)
// 2. Check token bucket → if denied, return 429
// 3. Process request → store in idempotency cache → return response
Bucket capacity = burst size. Refill rate = average rate. Idle periods accumulate tokens for burst spending.
| Decision | Choice | Why |
|---|---|---|
| Algorithm | Token bucket: capacity=500, refill_rate=100/sec | Allows instant burst of 500. After burst, limited to 100/sec until bucket refills. Average never exceeds 100/sec. |
| Capacity (burst size) | 500 tokens (5 seconds of accumulated capacity) | Matches batch settlement size. Merchant can send 500 at once after 5+ seconds of inactivity. |
| Refill rate | 100 tokens per second | This IS the average rate. After a burst, bucket refills at 100/sec. Sustained traffic cannot exceed this. |
| Implementation | Lazy refill: on each request, calculate tokens_added = elapsed_since_last × refill_rate | No background timer needed. O(1) per request. Store only: {token_count, last_request_time}. |
| Storage | Redis: HASH with tokens_remaining + last_refill_ts per API key | Atomic check-deduct via Lua. Shared across all app servers. |
| Headers | X-RateLimit-Limit: 100, X-RateLimit-Remaining: {tokens}, X-RateLimit-Burst: 500 | Client knows both their sustained limit and remaining burst capacity. |
| Failure | Impact | Recovery |
|---|---|---|
| Merchant sends burst larger than capacity | Requests beyond 500 get 429 immediately | Client-side batching: split into chunks of 500, wait refill time between chunks. Return Retry-After header with exact wait time. |
| Sustained traffic at exactly 100/sec (no idle) | Bucket never refills above 100, no burst available | Correct behavior. Sustained rate is the cap. If merchant needs burst + sustained, increase capacity (plan upgrade). |
| Clock drift in lazy refill | Incorrect token calculations | Use Redis TIME for consistent clock. Cap refill at capacity (prevent overflow on large time gaps). |
| Concurrent requests drain bucket simultaneously | Race condition: both see tokens=1, both deduct | Atomic Lua script: read + deduct + return new level in one operation. |
| Payment processing requires idempotency | Retried requests shouldn't double-charge | Idempotency keys bypass rate limiting for retries (same idempotency key = same request, don't deduct again). |
Key points for burst-allowing rate limiting