System Design Concepts

No fluff — visual, concise, interview-ready

🔢 14 · KEY NUMBERS

Latency Numbers Every Programmer Should Know

Jeff Dean's famous latency numbers — updated for 2024 hardware. These form the foundation of every back-of-envelope calculation.

OperationLatencyRelative ScaleNotes
L1 cache reference~1 ns1×CPU register-adjacent
Branch mispredict~3 ns3×Pipeline flush penalty
L2 cache reference~4 ns4×4× L1
L3 cache reference~12 ns12×Shared across cores
Mutex lock/unlock~17 ns17×Uncontended
Main memory (RAM)~100 ns100×DRAM access
Compress 1KB (Snappy)~3 µs3,000×Fast compression
Send 2KB over 1Gbps NIC~20 µs20,000×Network card latency
SSD random read~100 µs100,000×NVMe: ~20 µs
Read 1MB sequentially (RAM)~250 µs250,000×Memory bandwidth
Same datacenter RTT~500 µs500,000×Within AZ
Read 1MB sequentially (SSD)~1 ms1,000,000×Sequential is fast
HDD seek~10 ms10,000,000×Mechanical movement
Read 1MB sequentially (HDD)~20 ms20,000,000×After seek
Send packet CA→NL→CA~150 ms150,000,000×Speed of light limit
Key Insight: Memory is 100× faster than SSD, SSD is 100× faster than HDD, and local is 1000× faster than cross-continent. Design your hot path to stay in memory.

Visual Latency Scale (Logarithmic)

Latency Scale — Orders of Magnitude (log₁₀) 1ns 10ns 100ns 1µs 100µs 10ms 150ms L1 Cache — 1 ns L2 Cache — 4 ns RAM — 100 ns Compress 1KB — 3 µs SSD Random Read — 100 µs Datacenter RTT — 500 µs HDD Seek — 10 ms Cross-Region RTT — 150 ms

Network Latency Breakdown

What happens during a typical HTTPS request — each phase adds latency

DNS Lookup 20-120 ms (cached: 0 ms) TCP Handshake 1 RTT ~14-100 ms TLS Handshake 1-2 RTT ~28-200 ms HTTP Request 1 RTT ~14-100 ms Server Work 5-500 ms (app dependent) Total cold start: 80-1000+ ms | With keep-alive: 20-600 ms

Jeff Dean's Numbers (2024 Updated)

Operation20122024
L1 cache0.5 ns~1 ns
L2 cache7 ns~4 ns
RAM100 ns~80-100 ns
SSD random read150 µs~16-100 µs (NVMe)
HDD seek10 ms~4-10 ms
1 Gbps send 1MB10 ms~1 ms (10Gbps+)
Datacenter RTT500 µs~200-500 µs

Latency Rules of Thumb

🟢 Fast (<1ms): In-memory ops, cache hits, local IPC
🟡 Medium (1-10ms): DB queries, SSD reads, same-region calls
🟠 Slow (10-100ms): HDD, cross-AZ, external API calls
🔴 Very Slow (>100ms): Cross-region, cold starts, DNS lookups

Optimization Priority: Cache hot data → reduce network hops → batch I/O → move compute closer to data

Throughput Numbers

Capacity benchmarks for common systems — use these for back-of-envelope sizing. Numbers assume modern hardware (2024) with proper tuning.

SystemRead ThroughputWrite ThroughputNotes
PostgreSQL~10-30K reads/sec~1-5K writes/secSingle node, indexed queries
MySQL (InnoDB)~10-25K reads/sec~1-5K writes/secSimilar to Postgres
Redis~100K-1M ops/sec~100K-500K ops/secIn-memory, single-threaded per shard
Memcached~200K-1M ops/sec~200K-1M ops/secMulti-threaded, simple K/V
Cassandra~20-50K reads/sec/node~50-100K writes/sec/nodeWrite-optimized LSM tree
DynamoDBUnlimited (auto-scale)Unlimited (auto-scale)Per-partition: 3K RCU, 1K WCU
MongoDB~20-50K reads/sec~10-30K writes/secSingle node, WiredTiger
Kafka~1M+ msg/sec (consume)~1M+ msg/sec (produce)Per cluster, partitioned
RabbitMQ~20-50K msg/sec~20-50K msg/secPer node, persistent
Elasticsearch~10-50K queries/sec~10-50K docs/sec indexingDepends on query complexity
S3~5,500 GET/sec/prefix~3,500 PUT/sec/prefixPartition prefixes for more
NGINX / HAProxy~100K-500K concurrent connectionsEvent-driven, non-blocking
gRPC~50-100K RPC/sec per serverHTTP/2 multiplexing

Throughput Comparison (Relative Scale)

System Throughput — Operations per Second (log scale) Redis Memcached Kafka NGINX Cassandra gRPC Server Elasticsearch MongoDB PostgreSQL S3 (per prefix) 1M ops/s 1M ops/s 1M msg/s 500K conn 100K w/s 100K rpc/s 50K q/s 50K r/s 10-30K r/s 5.5K GET/s 1K 10K 100K 500K 1M+

Connections per Server

🟢 Event-Driven (epoll/kqueue)

RuntimeConnections
NGINX500K-1M
Node.js (libuv)100K-500K
Netty (Java)100K-500K
WebSocket server500K-1M

🟡 Lightweight Threads

RuntimeConcurrency
Go goroutines100K-1M
Erlang processes1M+
Java virtual threads100K-1M
Rust async (tokio)500K-1M

🔴 OS Threads

RuntimeThreads
Java (platform)5K-10K
Python (GIL)~1K effective
C++ threads10K-50K
Linux max~32K default
Scaling Rule: For WebSocket-heavy apps (chat, gaming, live updates), budget 500K connections per server with epoll. For request/response APIs, budget 10-50K RPS per server depending on processing complexity. Horizontal scaling = more servers, not bigger servers.

Storage & Size Estimation

Know the size of common data objects and storage tiers to quickly estimate capacity requirements.

Common Data Sizes

Data TypeTypical Size1M Records1B Records
Tweet / Short message~140 B (text only)~140 MB~140 GB
Chat message (with metadata)~500 B~500 MB~500 GB
User profile (basic)~1 KB~1 GB~1 TB
User profile (rich, with prefs)~5-10 KB~5-10 GB~5-10 TB
Email (avg with headers)~50 KB~50 GB~50 TB
Web page (HTML + assets)~2-5 MB~2-5 TB~2-5 PB
Photo (compressed JPEG)~200 KB - 5 MB~200 GB - 5 TB~200 TB - 5 PB
Photo (multiple resolutions)~1-10 MB total~1-10 TB~1-10 PB
Video (1 min, 720p)~50 MB~50 TB~50 PB
Video (1 min, 1080p)~130 MB~130 TB~130 PB
Log entry~200-500 B~200-500 MB~200-500 GB
Search index entry~500 B - 2 KB~500 MB - 2 GB~500 GB - 2 TB

Storage Hierarchy — Capacity, Speed & Cost

Storage Hierarchy — Speed vs Capacity vs Cost L1 64 KB | 1 ns | ~$$$$$ L2 256 KB-1 MB | 4 ns L3 8-64 MB | 12 ns RAM 16-512 GB | 100 ns | ~$5/GB SSD (NVMe) 1-30 TB | 100 µs | ~$0.10/GB HDD 4-20 TB | 10 ms | ~$0.02/GB Object Storage (S3) Unlimited | 50-100 ms | ~$0.023/GB/mo Archive (Glacier / Cold) Unlimited | hours | ~$0.004/GB/mo ← FASTER / COSTLIER ← LARGER / CHEAPER Rule: Each tier is ~10-100× slower but ~10-100× cheaper and larger than the one above Design principle: Keep hot data in upper tiers, cold data in lower tiers

Growth Estimation Template

📊 Daily → Monthly → Yearly

Given:
  Users = 10M DAU
  Actions/user/day = 5 posts
  Size per action = 1 KB (text) + 200 KB (image)

Daily:
  Posts = 10M × 5 = 50M posts/day
  Storage = 50M × 201 KB = ~10 TB/day

Monthly:
  Storage = 10 TB × 30 = ~300 TB/month

Yearly:
  Storage = 300 TB × 12 = ~3.6 PB/year

With replication (3×):
  Total = 3.6 PB × 3 = ~10.8 PB/year

📐 Quick Multipliers

PeriodSecondsApprox
1 minute60~60
1 hour3,600~4K
1 day86,400~100K
1 month2,592,000~2.5M
1 year31,536,000~30M

Memory Rule: 1 day ≈ 10⁵ sec, 1 year ≈ 3×10⁷ sec
Storage Rule: 2³⁰ = 1 GB, 2⁴⁰ = 1 TB, 2⁵⁰ = 1 PB

Back-of-Envelope Estimation

The most critical skill in system design interviews. Master these formulas and you can size any system in 5 minutes.

Core Estimation Formulas

═══ TRAFFIC ═══
Total Users → DAU = Total × DAU% (typically 10-30%)
Requests/day = DAU × actions_per_user
RPS (avg) = Requests/day ÷ 86,400 (seconds in a day)
RPS (peak) = RPS_avg × peak_multiplier (typically 3-5×)

═══ STORAGE ═══
Storage/day = Requests/day × avg_object_size
Storage/year = Storage/day × 365
With replication = Storage × replication_factor (typically 3×)
With overhead = Storage × 1.3 (indexes, metadata, fragmentation)

═══ BANDWIDTH ═══
Bandwidth (avg) = RPS × avg_response_size
Bandwidth (peak) = Peak_RPS × avg_response_size
Egress/month = Bandwidth_avg × seconds_in_month

═══ SERVERS ═══
Servers needed = Peak_RPS ÷ RPS_per_server
With headroom = Servers × 1.5 (for failover + growth)
Cache servers = Hot_data_size ÷ RAM_per_server

═══ CACHE ═══
Cache size = Working_set × hit_rate_target
Working set ≈ 20% of total data (80/20 rule)
Cache hit rate target: 95-99% for read-heavy systems

Estimation Framework Flowchart

Back-of-Envelope Estimation Framework Total Users e.g., 100M ×20% DAU 20M active/day ×10 req Requests/Day 200M req/day ÷100K Avg RPS ~2,300 RPS ×5 Peak RPS ~11,500 RPS Storage 200M × 1KB = 200GB/day = 73 TB/year Bandwidth 11.5K × 10KB = 115 MB/s = ~1 Gbps peak Servers 11.5K ÷ 1K RPS/srv = ~12 servers (+50%) ✓ Final Answer: 18 app servers, 73 TB storage/yr, 1 Gbps peak bandwidth Add: 3× replication, cache layer (Redis), CDN for static assets, DB read replicas

Common Estimation Patterns

SystemDAUActions/User/DayObject SizeDaily StoragePeak RPS
Social Media Post500M2 posts + 50 reads1 KB text + 500 KB media~500 TB~300K
Chat Message100M50 messages500 B~2.5 TB~250K
Video Upload50M0.01 uploads + 5 views500 MB (upload)~250 TB~15K uploads
Search Query1B5 searches~100 B query, 10 KB result~50 TB results~250K
E-commerce Order10M0.1 orders + 20 browses5 KB order, 50 KB page~10 TB~12K orders
Ride-sharing20M2 rides + 100 location pings200 B per ping~400 GB pings~50K

Powers of 10 — Quick Reference

Powers of 2 (Storage)

PowerValueNameReal-World
2¹⁰1,0241 KBA short paragraph
2²⁰~1M1 MBA book, a photo
2³⁰~1B1 GBA movie, 1K photos
2⁴⁰~1T1 TB1M photos, 500h video
2⁵⁰~1P1 PBNetflix library (3×)

Powers of 10 (Traffic)

ScaleRPSReal-World Analogy
10¹10 RPSPersonal blog
10²100 RPSSmall SaaS app
10³1K RPSMedium startup
10⁴10K RPSLarge app (Slack)
10⁵100K RPSTwitter, Netflix
10⁶1M RPSGoogle Search
Interview Tip: Always state your assumptions clearly. "I'll assume 100M total users, 20% DAU, 10 actions per user per day." Interviewers care more about your process than exact numbers. Round aggressively — 86,400 ≈ 100K is perfectly fine.
80/20 Rule: 20% of data generates 80% of traffic. Cache the hot 20% in Redis/Memcached. For a 1TB dataset, you need ~200GB of cache to serve 80% of reads from memory.
Rule of 72: Doubling time = 72 ÷ growth%. At 10% monthly growth, data doubles in ~7 months. At 5% monthly growth, doubles in ~14 months. Plan capacity for 2× current load minimum.

Cost Estimation

Cloud cost awareness separates senior engineers from juniors. Know these numbers to make architecture decisions that balance performance with budget.

AWS / GCP Cost Reference (2024 Pricing)

CategoryServiceSpecMonthly CostNotes
ComputeEC2 m5.xlarge4 vCPU, 16 GB~$140/moOn-demand, general purpose
EC2 m5.4xlarge16 vCPU, 64 GB~$560/moTypical app server
Lambda1M invocations~$0.20+ $0.0000167/GB-sec
Fargate1 vCPU, 2 GB~$30/moServerless containers
StorageS3 StandardPer TB~$23/TB/mo+ request costs
S3 GlacierPer TB~$4/TB/moRetrieval: hours
EBS (gp3)Per TB~$80/TB/moBlock storage for EC2
EFSPer TB~$300/TB/moShared file system
DatabaseRDS (db.r5.xlarge)4 vCPU, 32 GB~$350/moMulti-AZ: 2×
DynamoDBOn-demand$1.25/M writes, $0.25/M readsPay per request
ElastiCache (r6g.large)2 vCPU, 13 GB~$200/moRedis managed
NetworkData Transfer (egress)First 10 TB$0.09/GBIngress is free
CloudFront CDNFirst 10 TB$0.085/GBCheaper than direct
MessagingSQSPer 1M requests~$0.40Standard queue
MSK (Kafka)3 brokers (m5.large)~$500/moManaged Kafka

Monthly Cost for Common Architectures

💬 Chat App (10M DAU)

ComponentCost/mo
20 app servers (m5.xl)$2,800
WebSocket servers (10×)$1,400
Redis cluster (5 nodes)$1,000
RDS Multi-AZ$700
S3 (media, 50TB)$1,150
Bandwidth (100TB)$9,000
Total~$16K/mo

🎬 Video Platform (5M DAU)

ComponentCost/mo
Transcoding (GPU)$5,000
S3 storage (500TB)$11,500
CloudFront CDN (1PB)$40,000
App servers (10×)$1,400
Search (ES cluster)$2,000
Database (RDS)$1,500
Total~$61K/mo

🛒 E-commerce (2M DAU)

ComponentCost/mo
App servers (8×)$1,120
RDS Multi-AZ (large)$1,400
ElastiCache (3 nodes)$600
Elasticsearch$1,500
S3 + CloudFront$3,000
SQS + Lambda$200
Total~$8K/mo

Cost Optimization Strategies

💰 Reserved Instances

Savings: 30-60% vs on-demand
  • 1-year RI: ~30% savings
  • 3-year RI: ~60% savings
  • Best for: steady-state workloads
  • Commit: predictable base load

⚡ Spot Instances

Savings: 60-90% vs on-demand
  • Can be interrupted with 2-min notice
  • Best for: batch jobs, CI/CD, stateless workers
  • Not for: databases, user-facing servers
  • Use spot fleets for availability

📏 Right-Sizing

Savings: 20-40% typical
  • Monitor CPU/memory utilization
  • Target: 60-70% avg utilization
  • Downsize over-provisioned instances
  • Use auto-scaling for variable load
Cost Rule of Thumb: For most applications: Bandwidth > Compute > Storage. Video/media platforms spend 60-70% on CDN/bandwidth. CRUD apps spend 50-60% on compute. Data platforms spend 40-50% on storage. Always check: "Am I paying for idle resources?"

SLA Math & Availability

Availability is measured in "nines." Each additional nine is 10× harder to achieve and 10× more expensive. Know how to calculate composite SLAs.

Availability Levels

SLA LevelUptime %Downtime/YearDowntime/MonthDowntime/WeekTypical Systems
One Nine90%36.5 days72 hours16.8 hoursBatch jobs, internal tools
Two Nines99%3.65 days7.2 hours1.68 hoursPersonal projects
Three Nines99.9%8.76 hours43.8 min10.1 minBusiness SaaS apps
Four Nines99.99%52.6 min4.38 min1.01 minE-commerce, fintech
Five Nines99.999%5.26 min26.3 sec6.05 secPayment systems, DNS
Six Nines99.9999%31.5 sec2.63 sec0.6 secPacemakers, aviation

Serial vs Parallel Availability

Availability: Serial (worse) vs Parallel (better) Serial (all must work) A_total = A₁ × A₂ × A₃ Web Server 99.9% App Server 99.9% Database 99.9% Result: 0.999 × 0.999 × 0.999 = 99.7% (26.3 hrs/yr downtime) Parallel (any one works) A_total = 1 - (1-A₁)(1-A₂) Server A 99.9% Server B 99.9% Result: 1-(0.001)² = 99.9999% (31.5 sec/yr) Composite SLA Example: Typical 3-Tier Architecture Load Balancer 99.99% (HA) × App Servers (3×) 1-(1-0.999)³ = 99.9999% × DB Primary+Replica 1-(1-0.999)² = 99.9999% × Redis (HA) 99.99% Composite: 99.99% × 99.9999% × 99.9999% × 99.99% ≈ 99.98% (1.75 hrs/yr)

Error Budget Concept

📊 What is an Error Budget?

Error budget = allowed downtime based on SLO. It's the "budget" you can "spend" on deployments, experiments, and incidents.

SLOError Budget/MonthMeaning
99.9%43.8 minutes~1 incident allowed
99.95%21.9 minutesVery tight
99.99%4.38 minutesAlmost zero tolerance

🎯 SLO vs SLA vs SLI

SLI (Indicator): The metric you measure (e.g., latency p99 < 200ms)
SLO (Objective): Internal target (e.g., 99.95% requests < 200ms)
SLA (Agreement): External contract with penalties (e.g., 99.9% or credits)

Rule: SLO should be stricter than SLA. If SLA = 99.9%, set SLO = 99.95%. The gap is your safety buffer before you breach the contract.
Key Insight: Adding more serial components always reduces availability. A system with 5 serial components at 99.9% each = 99.5% overall (43.8 hrs/yr downtime). Solution: add redundancy (parallel) at each layer, especially the weakest link.

Interview Quick Reference — The 20 Numbers You Must Know

Your "cheat card" for system design interviews. Memorize these anchors and you can derive everything else.

⚡ Latency Anchors

#FactAnchor
1RAM access100 ns
2SSD random read100 µs
3HDD seek10 ms
4Same datacenter RTT0.5 ms
5Cross-region RTT150 ms

🚀 Throughput Anchors

#FactAnchor
6Redis ops/sec100K ops/s
7Kafka messages/sec1M msg/s
8PostgreSQL reads/sec10K reads/s
9WebSocket connections/server500K conn
10S3 GET per prefix5,500/s

💾 Storage Anchors

#FactAnchor
11Tweet / short msg~140 B
12User profile~1 KB
13Photo (JPEG)~200 KB - 5 MB
14Video (1 min, 720p)~50 MB
151 day in seconds~100K sec

💰 Cost & SLA Anchors

#FactAnchor
16S3 cost per TB/mo$23/TB
17EC2 m5.xlarge/mo~$140
18Bandwidth egress/GB$0.09/GB
1999.9% downtime/yr8.76 hours
2099.99% downtime/yr52.6 minutes

Quick Derivation Patterns

Interview Derivation: From Requirements to Architecture Pattern: RPS Calculation 100M users × 20% DAU = 20M 20M × 10 req/day = 200M/day 200M ÷ 100K sec = 2,000 RPS avg Peak = 2,000 × 5 = 10,000 RPS Pattern: Storage Sizing 50M posts/day × 1KB = 50 GB/day 50 GB × 365 = 18.25 TB/year With 3× replication = 55 TB/year 5-year plan: ~275 TB total Pattern: Server Count Peak RPS = 10,000 Each server handles ~1K RPS Need: 10 servers minimum With 50% headroom: 15 servers Pattern: Cache Sizing Total data: 18 TB Hot data (20%): 3.6 TB Redis node: 64 GB RAM Need: 3,600 ÷ 64 = 57 Redis nodes Pattern: Bandwidth Peak RPS = 10K, avg response = 50KB Bandwidth = 10K × 50KB = 500 MB/s = 4 Gbps peak outbound Monthly: ~1.3 PB egress Pattern: Quick Cost 15 servers × $140 = $2,100/mo 55 TB S3 × $23 = $1,265/mo 1.3 PB egress × $90/TB = $117K/mo → CDN needed! ($40K vs $117K)

Interview Mnemonics

🧠 "The 100× Rule"

Each storage tier is ~100× slower:

L1 → RAM: 100×
RAM → SSD: 1000×
SSD → HDD: 100×
HDD → Network: 15×

🧠 "The 1000× Rule"

Scaling units of data:

1 KB × 1M = 1 GB
1 KB × 1B = 1 TB
1 MB × 1M = 1 TB
1 MB × 1B = 1 PB

🧠 "The Time Rule"

Seconds in common periods:

1 day ≈ 10⁵ seconds
1 month ≈ 2.5 × 10⁶ sec
1 year ≈ 3 × 10⁷ sec
Peak = avg × 3-5
Final Interview Tip: When asked "Design X for Y million users," immediately write down: (1) DAU estimate, (2) RPS calculation, (3) Storage per year, (4) Bandwidth peak. This shows structured thinking and buys you time to think about the actual architecture. Round aggressively — interviewers want to see your process, not calculator precision.
Sanity Checks: After your estimation, verify: Is RPS reasonable for the system type? (Social media: 100K+, SaaS: 1-10K). Is storage growing faster than you can afford? (If >1PB/year, you need tiered storage). Is bandwidth cost dominant? (If yes, add CDN). Are you over-provisioning? (Target 60-70% utilization).