Multi-tenant rate limiting · Noisy neighbor protection

➔? Design multi-tenant rate limiting: per-tenant fairness + global capacity protection + hot-key handling for large tenants

Concepts Involved

Rate Limiting Load Balancer Redis Backpressure Sharding

Problem Statement

Design rate limiting for a multi-tenant SaaS platform where one tenant's traffic spike cannot degrade service quality for others. The system must enforce per-tenant limits, protect global shared capacity, and handle "whale" tenants (10-100x average traffic) without dedicated infrastructure per tenant.

Core challenge: The noisy neighbor problem: one tenant sends 100x normal traffic, saturating shared database connections, CPU, and memory. Other tenants experience latency spikes and timeouts. You can't provision dedicated infrastructure for every tenant (cost-prohibitive), so you need fair scheduling + admission control on shared resources.

1000+

tenants

shared infrastructure

Noisy Neighbor

1 tenant saturates

shared resources

Two-Layer

per-tenant + global

both must pass

Fair Share

weighted scheduling

proportional to plan

Functional Requirements

What the system must do · multi-tenant fairness and capacity protection

Must Have (Core)

1. Per-tenant admission control · each tenant limited to their plan's rate
2. Global capacity protection · total system throughput cannot exceed shared infra limits
3. Weighted fair queueing when at capacity · remaining throughput shared proportional to plan tier
4. Whale tenant detection + isolation · tenants exceeding 20% total throughput routed to isolated pool
5. Self-service usage dashboard · tenants see their own consumption and limits
6. Two distinct 429 error codes · tenant_limit_exceeded vs system_capacity_exceeded

Out of Scope (this design)

✗ Billing/invoicing (separate billing service)
✗ SLA enforcement/credits (business logic layer)
✗ Tenant provisioning (onboarding workflow)
✗ Data isolation (separate concern · row-level security)
✗ Per-endpoint limits within tenant (finer-grained throttling)

Non-Functional Requirements

Quality constraints shaping the multi-tenant rate limiting design

Property	Target	Why It Matters / Design Impact
Isolation	One tenant’s spike must NOT degrade others’ P99 latency	Per-tenant limits + whale isolation pool prevents noisy neighbor effect.
Capacity	100K req/sec total across all tenants	Shared infrastructure ceiling. Global limiter prevents aggregate overload.
Detection	Whale identified within 5 seconds	Tenant exceeding 20% total throughput triggers dynamic isolation routing.
Fairness	Proportional to plan tier when at capacity	Remaining throughput shared by weighted fair queueing. Enterprise > Pro > Free.
Adaptive	Limits tighten when load > 80%	Dynamic throttling prevents cascading failures during system stress.

Key tension: Per-Tenant Fairness vs. Global Capacity. Even if every tenant is within their individual limit, the sum of all tenants can exceed system capacity. Two layers are required: per-tenant (their fault) and global (system overloaded) · with different error codes for each.

Scale Estimation

Derive infrastructure sizing and isolation thresholds from tenant distribution

Given: 1000 tenants · System capacity 100K req/sec · Whale tenant can spike to 50K req/sec · 500 DB connections total

Step	What to Derive	Calculation	Result	Design Decision
1	Normal distribution	900 small (100 req/sec each) + 90 medium (2K each)	90K + 180K potential = exceeds capacity	Global limiter required even if each tenant is within their own limit
2	DB connection per-tenant max	500 total pool / 10 (safety factor)	50 connections max per tenant	Prevents single tenant from starving others of DB connections
3	Whale threshold	20% of 100K = 20K req/sec from single tenant	20K req/sec triggers isolation	Auto-route whale to dedicated worker pool when threshold exceeded
4	Isolated pool capacity	Reserve 30% of system capacity for whale overflow	30K req/sec isolated pool	Whale gets dedicated resources, shared pool protected for other 999 tenants
5	Redis counter memory	1000 tenants × 10 windows × 64 bytes per counter	~640KB for all rate counters	Trivial memory footprint. Redis handles easily with INCR + TTL.

Interview tip: Start by identifying the whale problem (step 3) · one tenant consuming 50% of capacity is the core challenge. Then show two-layer defense: per-tenant limits (their fault) + global capacity guard (system overloaded). The distinct error codes show maturity.

APIs

Rate limit enforcement interfaces and tenant monitoring

Rate Limit Middleware (All Requests)

Header: X-Tenant-Id identifies the tenant on every request
Layer 1 check: Per-tenant counter < plan limit? → pass or 429
Layer 2 check: Global counter < system capacity? → pass or 429
429 (tenant): {"error": "tenant_limit_exceeded", "retry_after": 1}
429 (system): {"error": "system_capacity_exceeded", "retry_after": 5}

Admin API: Tenant Monitoring

GET /admin/tenants/{id}/usage · Real-time request count, limit, % used
GET /admin/system/capacity · Global load %, active whale tenants
POST /admin/tenants/{id}/isolate · Manual whale isolation trigger
PUT /admin/tenants/{id}/limit · Override tenant rate limit

Data Model

Storage structures for per-tenant counters, global tracking, and isolation state

// Per-tenant rate counter (sliding window)
// Redis key: tenant:{id}:counter:{window_timestamp}
INCR tenant:acme:counter:1705312200     // Current window count
EXPIRE tenant:acme:counter:1705312200 60 // Auto-cleanup after window

// Global aggregate counter
INCR system:total:1705312200             // Total requests this window

// Whale tenant isolation flag
SET tenant:acme:isolated true EX 300     // Flag expires after 5 min
GET tenant:acme:isolated                 // Check before routing

// Tenant configuration (PostgreSQL)
CREATE TABLE tenants (
  tenant_id       VARCHAR(64) PRIMARY KEY,
  plan            VARCHAR(20),          -- free, pro, enterprise
  limit_rps       INT,                  -- requests per second limit
  burst_capacity  INT,                  -- token bucket burst size
  isolation_threshold FLOAT DEFAULT 0.2 -- % of system capacity
);

// Real-time metrics (Redis Sorted Set)
ZADD tenant:throughput 45000 "acme"     // Score = current req/sec
ZRANGEBYSCORE tenant:throughput 20000 +inf  // Find whales

Architecture · Two-Layer Rate Limiting

Layer 1: per-tenant admission control. Layer 2: global capacity protection. Both enforce independently.

Decision	Choice	Why
Layer 1	Per-tenant token bucket (capacity = plan limit)	Each tenant gets their contracted rate. Burst allowed up to bucket capacity.
Layer 2	Global capacity limiter (total system throughput cap)	Even if all tenants are within their individual limits, total cannot exceed system capacity.
Fairness	Weighted fair queueing when global limit is hit	If system is at 90% capacity, remaining capacity is shared proportionally to plan tier.
Hot tenant	Isolated processing lane for whale tenants	Tenant consuming > 20% of capacity gets routed to dedicated worker pool. Prevents shared-pool saturation.
Backpressure	Adaptive: tighten tenant limits when global load > 80%	Dynamic throttling: when system is stressed, temporarily reduce all tenant limits proportionally.
Visibility	Per-tenant usage dashboards + 429 breakdown by tenant	Self-service: tenants see their own usage, can upgrade before hitting limits.

Two-layer design: Request arrives → Layer 1 checks per-tenant limit (fast, Redis INCR) → if passes, Layer 2 checks global capacity (shared semaphore/counter) → if both pass, request is processed. If either fails, return 429 with appropriate Retry-After.

Whale tenant handling: Monitor per-tenant traffic volume in real-time. When a tenant exceeds 20% of total throughput, dynamically route their requests to an isolated worker pool. This pool has its own DB connection pool, preventing the whale from starving other tenants of connections.

Anti-patterns: Only per-tenant limits · 100 tenants each at 80% of their limit can still overwhelm the system (100×80% = 8000% of capacity). Only global limits · one fast tenant consumes all shared capacity. Equal limits for all plans · enterprise customers subsidize free tier abuse.

Real-world: AWS · per-account service quotas + regional capacity. Salesforce · API call limits per org per 24hr rolling window. Datadog · per-org ingest rate limits. Snowflake · virtual warehouse isolation for noisy tenants.

Resilience & Edge Cases

Failure	Impact	Recovery
Whale tenant sudden spike	Saturates shared DB connections before rate limiter kicks in	Connection pool per-tenant quotas. Admission control at connection checkout, not just request level.
All tenants spike simultaneously (event-driven)	Global capacity hit, everyone degraded equally	Priority tiers: enterprise tenants shed last. Auto-scale with pre-warming. Queue excess with fair ordering.
New tenant onboards with massive backfill	Initial data import floods the system	Onboarding rate limit (lower than steady-state). Dedicated import pipeline. Background job queue with throttling.
Rate limit counter storage failure	Cannot enforce per-tenant limits	Local approximate counters as fallback. Conservative local limit = global_capacity / active_tenants.
Tenant complaint about unfair limiting	Tenant claims they're within their plan but getting 429s	Distinguish per-tenant 429 (their fault) from global-capacity 429 (system overloaded). Different error codes/messages.

Interview Cheat Sheet

Key points for multi-tenant rate limiting

1. Two-layer design · per-tenant limit AND global capacity limit (both must pass)
2. Noisy neighbor isolation · whale tenants get dedicated worker pools when they exceed threshold
3. Weighted fair queueing · when at capacity, remaining throughput shared proportional to plan tier
4. Adaptive throttling · dynamically tighten all limits when system load > 80%
5. Connection-level quotas · rate limiting requests is not enough — also limit DB connections per tenant
6. Self-service visibility · tenants see their own usage dashboard + 429 breakdown

System Design Case Study