SystemDesign.website
Track 7 ยท Module 7.4 ยท Distributed Reliability, Rate Limiting & DevOps

Observability & Distributed Tracing

How Google, Uber, and Netflix diagnose latency spikes across 500+ microservices in seconds. Master Metrics (Prometheus), Structured Logs, OpenTelemetry Distributed Tracing (Spans & Context Propagation), and Google SRE SLI/SLO/SLA Error Budgets.

โฑ๏ธ Reading Time: ~28 min
๐ŸŽฏ Architecture: OpenTelemetry, W3C Trace Context, Prometheus, SRE Error Budgets
๐ŸŽฌ Format: Hospital Chart Analogy + Live Trace Flamegraph Lab

๐Ÿฅ 1. The Core Intuition: The Hospital Patient Health Chart

๐Ÿฉบ The Analogy: Vitals vs Doctor's Notes vs MRI Contrast Scan

Imagine monitoring a critically ill patient in a modern intensive care unit:

1. Metrics (Vitals Monitor): The bedside monitor tracks Heart Rate (98 bpm), Blood Pressure (120/80), and Oxygen Level. These numeric time-series numbers have fixed storage cost and immediately alert the nurses WHEN a crisis begins.

2. Logs (Doctor's Detailed Progress Notes): Timestamped text entries written in the chart ("Administered 50mg antibiotic at 10:14am"). Shows exactly WHAT specific discrete event occurred.

3. Distributed Tracing (MRI / X-Ray Contrast Scan): A radio-opaque dye injected into the bloodstream that travels from the heart to the brain, kidneys, and liver, measuring transit time through every organ. Shows exactly WHERE the bottleneck, obstruction, or artery blockage is across the entire body!

๐Ÿ’ฅ The #1 Microservices Debugging NightmareWhen a single user checkout hops across 12 services and takes 8.4 seconds instead of 100ms:
โ€ข Grepping 12 separate log files across 50 server pods is mathematically impossible.
โ€ข Distributed Tracing connects every step under a single global trace_id, generating a complete Flamegraph pinpointing the exact 8.1s unindexed SQL query!

๐Ÿ“Š 2. Deep Dive: The 3 Pillars of Observability (M.E.L.T.)

1Metrics (Prometheus & Grafana)

Aggregated numeric measurements stored as time-series. Storage cost is constant $O(1)$ regardless of whether your system handles 10 QPS or 1,000,000 QPS!
โ€ข Counters: Monotonically increasing values (e.g. http_requests_total).
โ€ข Gauges: Values that go up and down (e.g. memory_usage_bytes).
โ€ข Histograms: Latency distribution buckets used to calculate p50, p95, and p99 percentiles.

2Structured Logs (Elasticsearch / Grafana Loki)

Discrete, contextual JSON events. Every production log must include structured fields:{"timestamp": "2026-08-30T12:00:00Z", "level": "ERROR", "trace_id": "4bf92f3577b34da6", "user_id": 9841, "msg": "Payment failed: Card declined"}

3Distributed Traces (OpenTelemetry & Jaeger)

Tracks the execution path of a single transaction across network boundaries.
โ€ข Trace: The entire end-to-end tree of a request.
โ€ข Span: A single named contiguous block of work (e.g. POST /checkout, SELECT FROM inventory).

๐ŸŒ 3. W3C Trace Context & Distributed Header Propagation

How does a downstream service know it is part of an ongoing trace? Microservices inject the standardized W3C Traceparent HTTP Header:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
โ€ข 00: Version number
โ€ข 4bf92f3577b34da6a3ce929d0e0e4736: Global 128-bit Trace ID (Shared across all 12 services)
โ€ข 00f067aa0ba902b7: 64-bit Parent Span ID (Identifies calling service)
โ€ข 01: Trace Flags (01 = Sampled / Recorded)

๐ŸŽฏ 4. Google SRE Reliability Math: SLI, SLO, SLA & Error Budgets

How do engineering teams balance shipping new features with system stability?

1. SLI (Indicator)

What you measure:(Good Requests / Total) = 99.94%

2. SLO (Objective)

Internal target goal:99.9% Uptime / 30 Days

3. SLA (Agreement)

Legal contract with penalty:99.5% (Refunds if breached)

๐Ÿ”ฅ The Error Budget Rule: (100% - SLO)At a 99.9% SLO, your team has an allowable downtime budget of 0.1% = 43.8 minutes per month.
If a buggy release causes 30 minutes of outage, your error budget is depleted. Feature deployments are frozen, and the entire team shifts 100% of engineering bandwidth to reliability and automated testing!

๐Ÿ’ป 5. Production OpenTelemetry Instrumentation (Multi-Language)

Here is the production implementation of OpenTelemetry span creation and context propagation across 5 languages:

Select Language:
# Python OpenTelemetry Manual Span Tracing
from opentelemetry import trace

tracer = trace.get_tracer("checkout-service")

async def process_checkout(order_id: str, amount: float):
    with tracer.start_as_current_span("process_checkout") as span:
        span.set_attribute("order.id", order_id)
        span.set_attribute("order.amount", amount)

        # Child span for DB query
        with tracer.start_as_current_span("inventory.reserve_sql") as db_span:
            db_span.set_attribute("db.system", "postgresql")
            await execute_sql("SELECT * FROM inventory WHERE id = %s", order_id)

        return {"status": "SUCCESS"}

๐Ÿงช 6. Interactive Lab: Live OpenTelemetry Flamegraph & Bottleneck Detector

โšก Real-Time Distributed Flamegraph Engine

OpenTelemetry Trace Visualizer (Jaeger Style)

Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736
Database Optimization:โŒ Missing Index (Table Scan)
SERVICE & OPERATION SPANDURATION WATERFALL
๐ŸŒ api-gateway: POST /checkout8,420 ms
๐Ÿ”‘ auth-service: verify_jwt18 ms
๐Ÿ“ฆ order-service: create_order8,390 ms
๐Ÿ’ณ payment-service: charge_stripe240 ms
๐Ÿ”ด inventory-db: SELECT * FROM inventory FOR UPDATE8,120 ms
๐Ÿšจ CRITICAL BOTTLENECK DETECTED: inventory-db took 8,120ms (96.4% of total trace) due to a missing index causing a sequential table scan across 15M rows. Click "โšก Add Database Index" to fix!

๐ŸŽฏ 7. Test Your Understanding

Scenario 1: p99 Tail Latency Investigation

Your payment service has an average response time of 50ms, but p99 latency is 4.5 seconds. Why are average metrics useless for detecting this problem, and which observability tool should you use?

Scenario 2: Google SRE Error Budget Depletion

Your team has a 99.9% SLO for checkout uptime. A bad deployment causes an outage that burns 100% of your monthly Error Budget in 2 weeks. According to Google SRE principles, what should happen next?

โ—‹

Completed this module?

Mark as finished to update your curriculum progress tracker.