SystemDesign.website
Track 7 ยท Module 7.2 ยท Distributed Reliability, Rate Limiting & DevOps

Fault Tolerance & Circuit Breakers

How Netflix, Amazon, and Uber survive partial outages without crashing their entire infrastructure. Master Circuit Breakers (Closed, Open, Half-Open), Exponential Backoff with Jitter, Bulkheads, and Load Shedding.

โฑ๏ธ Reading Time: ~28 min
๐ŸŽฏ Architecture: Circuit Breakers, Hystrix, Backoff + Jitter, Bulkheads
๐ŸŽฌ Format: Home Breaker Analogy + Live Pipeline Simulator

โšก 1. The Core Intuition: The Home Electrical Circuit Breaker

๐Ÿ”Œ The Analogy: The Basement Electrical Switch

Imagine your house electrical wiring:
If a faulty toaster shorts out in your kitchen, you don't want the electric current to overheat the wires and burn your entire house down.

Instead, an electrical Circuit Breaker in your basement trips instantly, cutting power to the kitchen while keeping your refrigerator, lights, and home office running safely.

In distributed microservices, when a downstream service slows down or fails, software Circuit Breakers trip open to prevent thread pool exhaustion and cascading system collapse!

๐Ÿ’ฅ How Cascading Failures Kill MicroservicesIf Service A calls Service B with a 10-second timeout, and Service B slows down:
1. Service A keeps worker threads blocked waiting for B.
2. In seconds, Service A runs out of available threads (Thread Exhaustion).
3. The API Gateway calling Service A also runs out of sockets.
4. Result: The entire multi-million dollar platform collapses because of 1 slow recommendation query!

๐Ÿ”„ 2. The 3 Circuit Breaker States (State Machine Anatomy)

The standard Circuit Breaker (Michael Nygard / Netflix Hystrix / Resilience4j) operates in 3 distinct states:

1CLOSED (Healthy Normal Flow)

All requests pass freely through to the downstream service. The breaker records successes, failures, and timeouts in a rolling window. If the failure rate exceeds the threshold (e.g. > 50% failures), the breaker TRIPS to OPEN.

2OPEN (Tripped / Fail-Fast Fallback)

The downstream service is never called (0 network sockets or worker threads wasted). Requests fail fast in < 0.5ms or immediately return a Graceful Fallback (cached data, default response). A cooldown sleep timer begins (e.g. 10 seconds).

3HALF-OPEN (Trial Probe)

Once the cooldown timer expires, the breaker enters Half-Open and allows a tiny sample of trial requests (e.g. 3 probe calls).
โ€ข If all probe calls succeed → Breaker RESETS to CLOSED.
โ€ข If any probe call fails → Breaker TRIPS back to OPEN for another cooldown cycle.

๐ŸŽฒ 3. Exponential Backoff with Full Jitter (Preventing Retry Storms)

When a database recovers from an outage, naive retries trigger a Retry Storm (Self-Inflicted DDoS):

โŒ Naive Fixed Retries (Retry Storm)

If 10,000 clients all retry at exact 1s, 2s, and 4s intervals, massive synchronized spikes hit the database simultaneously, knocking it down repeatedly.

โœ… Exponential Backoff with Full Jitter

Randomize the retry delay evenly between 0 and the backoff cap. Spreads 10,000 retries smoothly across time like background noise!

sleep = random(0, min(Cap, Base * 2^attempt))

๐Ÿšข 4. Bulkhead Pattern & Load Shedding

1. The Bulkhead Pattern

Just as a submarine has watertight doors so a leak in one compartment doesn't sink the ship:

Allocate isolated thread/connection pools per dependency. If the Recommendation service pool is 100% full, the Payment service still has its own dedicated 20 threads!

2. Load Shedding & Graceful Degradation

When CPU utilization exceeds 90%:

Drop non-essential traffic (e.g. animated avatars, recommendation feeds) with instant HTTP 503, preserving 100% compute capacity for core revenue transactions (checkout, billing).

๐Ÿ’ป 5. Production Circuit Breaker Implementation (Multi-Language)

Here is the production implementation of a 3-State Circuit Breaker with Fallback handling across 5 languages:

Select Language:
# Python Circuit Breaker with 3 States & Fallback
import time
from enum import Enum

class State(Enum):
    CLOSED = "CLOSED"
    OPEN = "OPEN"
    HALF_OPEN = "HALF_OPEN"

class CircuitBreaker:
    def __init__(self, failure_threshold=3, recovery_time=10):
        self.state = State.CLOSED
        self.failure_count = 0
        self.failure_threshold = failure_threshold
        self.recovery_time = recovery_time
        self.last_failure_time = 0

    def call(self, func, fallback_func):
        now = time.time()
        if self.state == State.OPEN:
            if now - self.last_failure_time > self.recovery_time:
                self.state = State.HALF_OPEN
            else:
                return fallback_func() # Fail-Fast in 0.2ms!

        try:
            result = func()
            if self.state == State.HALF_OPEN:
                self.state = State.CLOSED
                self.failure_count = 0
            return result
        except Exception as e:
            self.failure_count += 1
            self.last_failure_time = now
            if self.failure_count >= self.failure_threshold:
                self.state = State.OPEN
            return fallback_func()

๐Ÿงช 6. Interactive Lab: Live Circuit Breaker & Cascading Failure Simulator

โšก Real-Time Fault-Tolerance Engine

Circuit Breaker & Cascading Outage Lab

Inject downstream latency/errors and watch the breaker trip to protect threads!
1. Downstream Health:
2. Single Request:
3. Traffic Storm:
๐Ÿ”Œ Breaker StateCLOSED
Failures: 0 / 3
Cooldown: Inactive
โฑ๏ธ Observed Latency
20 ms
Live Downstream Call
๐Ÿ“Š Traffic Stats
Direct Calls: 0
Fallback Served: 0
๐Ÿš€ Circuit Breaker Active (Threshold = 3 failures): Downstream Inventory Service is healthy. Click "Send 1 Request" to start!

๐ŸŽฏ 7. Test Your Understanding

Scenario 1: Preventing Cascading Thread Exhaustion

Your Recommendation API experiences a 10-second database lock. Within 30 seconds, the main Checkout API also crashes, even though the Checkout database is 100% healthy. Why?

Scenario 2: Exponential Backoff & Full Jitter

Why does AWS recommend adding Full Jitter to exponential backoff algorithms during high-concurrency database recoveries?

โ—‹

Completed this module?

Mark as finished to update your curriculum progress tracker.