13. Observability — System Design Concepts

📊 13 · OBSERVABILITY

Logging

Structured JSON logs with correlation IDs. Levels: DEBUG → INFO → WARN → ERROR → FATAL. The foundation of observability — immutable forensic records of system behavior.

▸ Log Pipeline Architecture

Guarantees: Immutability (forensics, compliance). Searchability. Correlation IDs trace flow across services. Never log passwords/PII. Tools: ELK, Splunk, Loki, CloudWatch. Sample: 100% errors, 1-10% normal.

▸ Structured Logging Format (JSON)

{
  "timestamp": "2024-03-15T10:23:45.123Z",
  "level": "ERROR",
  "service": "payment-service",
  "version": "2.4.1",
  "trace_id": "abc123def456",
  "span_id": "span-789",
  "correlation_id": "order-98765",
  "message": "Payment processing failed",
  "metadata": {
    "user_id": "u-12345",
    "amount": 99.99,
    "currency": "USD",
    "error_code": "GATEWAY_TIMEOUT",
    "retry_count": 2
  },
  "context": {
    "host": "pod-payment-7b4f9",
    "region": "us-east-1",
    "environment": "production"
  }
}

▸ Log Levels Decision Guide

Level	When to Use	Example	Alert?
FATAL	System cannot continue — process will exit	Cannot bind to port, OOM killer	Page immediately
ERROR	Operation failed — needs attention	Payment gateway timeout, DB connection lost	Alert if rate > threshold
WARN	Unexpected but recoverable — degraded state	Retry succeeded, cache miss fallback, deprecated API call	Monitor trend
INFO	Normal business events — audit trail	Order placed, user login, deployment started	No
DEBUG	Developer troubleshooting — verbose	SQL query, request/response body, cache key	No — never in prod
TRACE	Extremely detailed — method entry/exit	Function arguments, loop iterations	No — dev only

▸ Log Sampling Strategies

Strategy	How It Works	Sample Rate	Best For
Always sample errors	100% of ERROR/FATAL logs retained	100%	All environments
Rate-based	Keep 1 in N logs per time window	1-10%	High-volume INFO logs
Priority-based	Sample based on log level + service criticality	Varies	Cost optimization
Tail-based	Decide after request completes (keep if slow/error)	Dynamic	Distributed tracing correlation
Hash-based	Consistent sampling by trace_id hash	Configurable	Correlated log/trace sampling

⚠️ Log Anti-Patterns: Logging PII/secrets · Unstructured string concatenation · Logging inside tight loops · No correlation ID · Logging full request bodies in prod · Missing timestamps

One-liner: Emit structured JSON with correlation_id + trace_id. Ship via Fluentd → Kafka → ES. Sample 100% errors, 1-10% info. Never log PII. Use log levels as severity contracts.

Metrics

Golden Signals (Google SRE): Latency · Traffic · Errors · Saturation — numeric time-series data for alerting and trending

▸ Metrics Collection Flow

Signal	What It Measures	Key Metrics	Alert When
Latency	Time to serve a request	p50 · p95 · p99 · p999. Never use avg — hides tail latency	p99 > SLA threshold
Traffic	Demand on the system	RPS · TPS · bytes/sec · active connections	Sudden spike or unexpected drop
Errors	Rate of failed requests	Error rate % · 5xx count · timeout count · DLQ depth	Error rate > 1% or any 5xx spike
Saturation	How full / overloaded the system is	CPU % · Memory % · Disk I/O · Thread pool queue depth	CPU > 80% sustained

Method	Full Form	Best For	Metrics
RED	Rate · Errors · Duration	Microservices, APIs	Requests/sec · Error % · p99 response time
USE	Utilization · Saturation · Errors	Infrastructure — CPU, memory, disk, network	CPU % · queue depth · device error count

▸ Why p99 Matters More Than Average

▸ SLI / SLO / SLA Relationship

SLI / SLO / SLA: SLI (indicator) — actual measured metric. SLO (objective) — internal target. SLA (agreement) — contractual commitment with penalty. SLO should be stricter than SLA. Error Budget = 100% − SLO.

⚠️ Cardinality Explosion: Every unique combination of label values creates a new time series. user_id as a label = millions of series → OOM. request_path with IDs (/users/12345) = unbounded. Fix: use bounded labels (method, status_code, service) and move high-cardinality to logs/traces. Rule of thumb: <10 values per label.

Tools: Prometheus — scrape + store time-series. Grafana — dashboards + alerting. Datadog — managed APM. CloudWatch — AWS native. OpenTelemetry — vendor-neutral standard. Thanos/Cortex — long-term Prometheus storage.

One-liner: Track p99 not average — use RED for services, USE for infra, alert on Golden Signals, define SLOs stricter than SLA. Watch for cardinality explosion.

Distributed Tracing

Track a single request across services. Trace ID → Spans → Parent-child relationships → Latency breakdown. Essential for debugging microservices.

▸ Trace / Span Hierarchy — Request Flow

Core Concepts: Trace — end-to-end journey of a request. Span — single unit of work (one service call). Root Span — first span, sets trace duration. Child Span — nested operation within parent. Span Context — trace_id + span_id + flags propagated across boundaries.

▸ Context Propagation Standards

Standard	Header Format	Example	Used By
W3C TraceContext	`traceparent` / `tracestate`	`00-abc123-span456-01`	OpenTelemetry (default), modern systems
B3 (Zipkin)	`X-B3-TraceId` / `X-B3-SpanId`	`X-B3-TraceId: abc123`	Zipkin, Spring Cloud Sleuth
B3 Single	`b3` (single header)	`abc123-span456-1`	Zipkin compact format
Jaeger	`uber-trace-id`	`abc123:span456:0:1`	Jaeger native (legacy)
Baggage	`baggage`	`user_id=123,region=us`	W3C Baggage — custom context across services

▸ Sampling Strategies: Head-Based vs Tail-Based

Strategy	Decision Point	Pros	Cons	Best For
Head-based	At trace start (first service)	Simple, low overhead, consistent	May miss interesting traces (errors decided later)	High-volume, cost-sensitive
Tail-based	After trace completes (collector)	Keep all errors/slow traces, smarter	Higher memory (buffer full traces), complex	Debug-focused, error analysis
Priority-based	Based on attributes (user tier, endpoint)	VIP users always traced	Configuration complexity	Multi-tenant systems
Rate-limiting	N traces per second per service	Predictable cost	May miss bursts	Budget-constrained

Async Boundary Propagation: For Kafka — inject trace context into message headers (traceparent header). For SQS — use message attributes. For gRPC — metadata headers. For HTTP — request headers. Key: consumer creates a new span linked to producer span (SpanLink, not parent-child).

Tools: OpenTelemetry (vendor-neutral SDK) · Jaeger (open-source backend) · Zipkin (lightweight) · Datadog APM · AWS X-Ray · Tempo (Grafana, uses object storage). Sample: 100% errors, 1-5% successes.

One-liner: Propagate W3C TraceContext headers across all boundaries. Use tail-based sampling to capture errors. Every span needs: service name, operation, duration, status, and parent span ID.

Monitoring & Alerting

Collect → Evaluate → Alert → Route → Respond → Root Cause Analysis. Only page if human action is needed.

▸ Alerting Pipeline Architecture

▸ Alert Quality: Good vs Bad

Quality	Alert Example	Why	Action
✅ Good	Error rate > 1% for 5 min on payment-svc	Sustained, actionable, specific service	Investigate payment gateway
✅ Good	p99 latency > 2s for 3 min	User-impacting, time-windowed	Check DB queries, scale pods
✅ Good	Error budget burn rate > 14.4x for 1h	SLO-based, predictive	Will breach SLO in 6h if unchecked
❌ Bad	Any single 500 error	Too noisy — single errors are normal	Alert fatigue, ignored
❌ Bad	CPU > 50%	Not actionable — 50% is healthy	Wastes on-call time
❌ Bad	Disk usage > 80% (no duration)	No time window — transient spikes trigger	False positives

▸ On-Call Best Practices

On-Call Rules: Every alert must have a runbook — link in alert annotation. Escalation policy — if no ack in 5 min, escalate. Rotation — weekly, with handoff notes. Blameless culture — focus on systems, not people. Toil budget — if >50% time on toil, fix the system.

▸ Burn Rate Alerting (Multi-Window)

Burn Rate Concept: If SLO = 99.9% over 30 days → error budget = 0.1% = 43.2 min of downtime/month.

Burn rate 1x = consuming budget at exactly the allowed rate (will exhaust in 30 days).
Burn rate 14.4x = will exhaust entire budget in ~2 days → page immediately.
Burn rate 6x = will exhaust in ~5 days → ticket, investigate.
Burn rate 1x = normal → no action.

Multi-window: Use short window (5 min) to detect fast burns + long window (1h) to confirm sustained. Prevents false positives from brief spikes.

Tools: PagerDuty · Opsgenie · Grafana OnCall · Prometheus Alertmanager. Key features: deduplication, grouping, silencing, inhibition, escalation chains.

One-liner: Only page if human action needed. Use burn rate alerts over static thresholds. Every alert needs a runbook. Group + deduplicate to prevent alert fatigue.

OpenTelemetry

Vendor-neutral observability framework — one SDK for Traces + Metrics + Logs. Instrument once, export to any backend. CNCF project, industry standard.

▸ OpenTelemetry Architecture

▸ Three Signals Unified

Signal	What It Captures	OTel API	Data Model	Best Backend
Traces	Request flow across services	TracerProvider → Tracer → Span	Spans with parent-child, attributes, events	Jaeger, Tempo, Zipkin
Metrics	Numeric measurements over time	MeterProvider → Meter → Instrument	Counter, Histogram, Gauge, UpDownCounter	Prometheus, Datadog
Logs	Discrete events with context	LoggerProvider → Logger → LogRecord	Timestamp, severity, body, attributes, trace_id	Loki, Elasticsearch

▸ Instrumentation Patterns

Type	How	Effort	Coverage	Example
Auto-instrumentation	Agent/SDK hooks into frameworks automatically	Zero code changes	HTTP, DB, gRPC, messaging	Java agent, Python auto-instr, Node.js require hook
Manual instrumentation	Developer adds spans/metrics in code	Code changes required	Business logic, custom operations	`tracer.startSpan("processPayment")`
Semantic Conventions	Standardized attribute names	Follow naming guide	Cross-service consistency	`http.method`, `db.system`, `rpc.service`

▸ Collector Deployment Patterns

Pattern	Architecture	Pros	Cons	Best For
Sidecar	Collector per pod (DaemonSet or sidecar container)	Isolation, per-service config, low latency	More resource usage, many instances	Kubernetes, per-service sampling
Gateway	Centralized collector cluster (Deployment)	Fewer instances, centralized config, easier to manage	Single point of failure, network hop	Simple setups, centralized processing
Agent + Gateway	Local agent → central gateway (two-tier)	Best of both: local buffering + central processing	More complex, two configs	Large-scale production

Key Benefits: Vendor-neutral — switch backends without re-instrumenting. Correlation — trace_id links logs ↔ traces ↔ metrics. Semantic conventions — standardized attribute names across languages. Collector — offloads processing from app (batching, retry, sampling).

One-liner: Use OTel auto-instrumentation for 80% coverage, add manual spans for business logic. Deploy collector as agent + gateway in production. Correlate all three signals via trace_id.

Dashboards & Visualization

Dashboards are the first thing you look at during an incident. Design for clarity: USE for infra, RED for services, business KPIs for stakeholders.

▸ Dashboard Design Principles

Principle	Description	Example
Audience-first	Different dashboards for different roles	SRE: latency/errors. PM: conversion rate. Exec: uptime %
Top-down drill	Start high-level, click to detail	Service overview → endpoint breakdown → individual trace
Time alignment	All panels share same time range	Correlate CPU spike with latency increase at same moment
Annotations	Mark deployments, incidents on timeline	Vertical line: "v2.4.1 deployed" — correlate with metric change
Thresholds visible	Show SLO/SLA lines on graphs	Red line at p99 = 500ms (SLO boundary)
≤ 8 panels	Cognitive load limit per dashboard	More panels = slower load + harder to scan during incident

▸ The 4 Golden Dashboards Every Service Needs

1. Service Health (RED)

Request rate (RPS)
Error rate (%)
p50 / p95 / p99 latency
Active connections

2. Infrastructure (USE)

CPU utilization %
Memory usage / limits
Disk I/O + space
Network in/out

3. Dependencies

DB query latency
Cache hit ratio
External API latency
Queue depth / lag

4. Business KPIs

Orders/min
Signup conversion
Revenue per minute
Active users

▸ Dashboard Anti-Patterns

Anti-Pattern	Problem	Fix
Too many panels	Cognitive overload, slow rendering, can't find signal in noise	Max 6-8 panels per dashboard. Split into sub-dashboards.
Vanity metrics	"Total requests ever" — not actionable, always goes up	Show rate of change (RPS), not cumulative totals
No context	Graph shows spike but no reference for "normal"	Add SLO threshold lines + deployment annotations
Average-only	Hides tail latency — 1% of users suffering	Always show p50 + p95 + p99 together
Stale dashboards	Panels for decommissioned services, broken queries	Quarterly dashboard review. Delete unused panels.
No drill-down	See problem but can't investigate further	Link panels to detailed dashboards, traces, logs

Grafana Tips: Use variables for service/environment selectors. Annotations for deploys. Alert rules directly on panels. Exemplars — click a metric data point to jump to the exact trace. Explore mode for ad-hoc investigation.

One-liner: Build 4 dashboards per service (RED, USE, Dependencies, Business). Max 8 panels each. Show percentiles not averages. Add SLO lines + deploy annotations. Link to traces for drill-down.

Incident Response

Structured process: Detect → Triage → Mitigate → Resolve → Postmortem. Goal: minimize user impact, learn from failures, prevent recurrence.

▸ Incident Lifecycle

▸ Severity Levels

Severity	Impact	Response Time	Who's Paged	Example
SEV1 — Critical	Complete outage, data loss, security breach	< 5 min	On-call + manager + exec	Payment system down, data breach
SEV2 — Major	Significant degradation, partial outage	< 15 min	On-call + team lead	50% of requests failing, one region down
SEV3 — Minor	Limited impact, workaround available	< 4 hours	On-call (next business day OK)	Non-critical feature broken, slow queries
SEV4 — Low	Cosmetic, no user impact	Next sprint	Ticket only	Typo in error message, minor UI glitch

▸ Key Reliability Metrics

Metric	Full Name	Formula	Target	How to Improve
MTTD	Mean Time to Detect	Time from failure start → alert fires	< 5 min	Better monitoring, anomaly detection, synthetic probes
MTTR	Mean Time to Resolve	Time from detection → service restored	< 30 min (SEV1)	Runbooks, automation, rollback capability, feature flags
MTBF	Mean Time Between Failures	Total uptime ÷ number of failures	Maximize	Chaos engineering, testing, redundancy, code quality
MTTA	Mean Time to Acknowledge	Time from alert → human acknowledges	< 5 min	Clear escalation policy, on-call rotation

▸ Postmortem Template Outline

Blameless Postmortem Structure:

1. Summary — What happened, impact, duration, severity
2. Timeline — Minute-by-minute: detection, actions taken, resolution
3. Root Cause — The "5 Whys" analysis. What actually broke and why
4. Impact — Users affected, revenue lost, SLO budget consumed
5. What Went Well — Detection speed, team response, tools that helped
6. What Went Wrong — Gaps in monitoring, slow detection, missing runbooks
7. Action Items — Specific, assigned, time-bound fixes (with owners + deadlines)
8. Lessons Learned — Systemic improvements, process changes

Incident Response Tools: PagerDuty (alerting + escalation) · Statuspage (external communication) · Slack/Teams (war room) · Jira/Linear (action items) · Rootly/incident.io (incident management) · Blameless (postmortem tracking)

Chaos Engineering: Proactively inject failures to find weaknesses before production incidents. Tools: Chaos Monkey (random instance kill) · Litmus (Kubernetes chaos) · Gremlin (managed platform). Start with Game Days — scheduled, controlled experiments with rollback plan.

One-liner: Detect fast (MTTD < 5 min), mitigate first (rollback > debug), resolve properly, then blameless postmortem with action items. Every SEV1 gets a postmortem within 48h.