Fault Tolerance & Circuit Breakers
How Netflix, Amazon, and Uber survive partial outages without crashing their entire infrastructure. Master Circuit Breakers (Closed, Open, Half-Open), Exponential Backoff with Jitter, Bulkheads, and Load Shedding.
โก 1. The Core Intuition: The Home Electrical Circuit Breaker
Imagine your house electrical wiring:
If a faulty toaster shorts out in your kitchen, you don't want the electric current to overheat the wires and burn your entire house down.
Instead, an electrical Circuit Breaker in your basement trips instantly, cutting power to the kitchen while keeping your refrigerator, lights, and home office running safely.
In distributed microservices, when a downstream service slows down or fails, software Circuit Breakers trip open to prevent thread pool exhaustion and cascading system collapse!
1. Service A keeps worker threads blocked waiting for B.
2. In seconds, Service A runs out of available threads (Thread Exhaustion).
3. The API Gateway calling Service A also runs out of sockets.
4. Result: The entire multi-million dollar platform collapses because of 1 slow recommendation query!
๐ 2. The 3 Circuit Breaker States (State Machine Anatomy)
The standard Circuit Breaker (Michael Nygard / Netflix Hystrix / Resilience4j) operates in 3 distinct states:
All requests pass freely through to the downstream service. The breaker records successes, failures, and timeouts in a rolling window. If the failure rate exceeds the threshold (e.g. > 50% failures), the breaker TRIPS to OPEN.
The downstream service is never called (0 network sockets or worker threads wasted). Requests fail fast in < 0.5ms or immediately return a Graceful Fallback (cached data, default response). A cooldown sleep timer begins (e.g. 10 seconds).
Once the cooldown timer expires, the breaker enters Half-Open and allows a tiny sample of trial requests (e.g. 3 probe calls).
โข If all probe calls succeed → Breaker RESETS to CLOSED.
โข If any probe call fails → Breaker TRIPS back to OPEN for another cooldown cycle.
๐ฒ 3. Exponential Backoff with Full Jitter (Preventing Retry Storms)
When a database recovers from an outage, naive retries trigger a Retry Storm (Self-Inflicted DDoS):
If 10,000 clients all retry at exact 1s, 2s, and 4s intervals, massive synchronized spikes hit the database simultaneously, knocking it down repeatedly.
Randomize the retry delay evenly between 0 and the backoff cap. Spreads 10,000 retries smoothly across time like background noise!
sleep = random(0, min(Cap, Base * 2^attempt))๐ข 4. Bulkhead Pattern & Load Shedding
Just as a submarine has watertight doors so a leak in one compartment doesn't sink the ship:
Allocate isolated thread/connection pools per dependency. If the Recommendation service pool is 100% full, the Payment service still has its own dedicated 20 threads!
When CPU utilization exceeds 90%:
Drop non-essential traffic (e.g. animated avatars, recommendation feeds) with instant HTTP 503, preserving 100% compute capacity for core revenue transactions (checkout, billing).
๐ป 5. Production Circuit Breaker Implementation (Multi-Language)
Here is the production implementation of a 3-State Circuit Breaker with Fallback handling across 5 languages:
# Python Circuit Breaker with 3 States & Fallback import time from enum import Enum class State(Enum): CLOSED = "CLOSED" OPEN = "OPEN" HALF_OPEN = "HALF_OPEN" class CircuitBreaker: def __init__(self, failure_threshold=3, recovery_time=10): self.state = State.CLOSED self.failure_count = 0 self.failure_threshold = failure_threshold self.recovery_time = recovery_time self.last_failure_time = 0 def call(self, func, fallback_func): now = time.time() if self.state == State.OPEN: if now - self.last_failure_time > self.recovery_time: self.state = State.HALF_OPEN else: return fallback_func() # Fail-Fast in 0.2ms! try: result = func() if self.state == State.HALF_OPEN: self.state = State.CLOSED self.failure_count = 0 return result except Exception as e: self.failure_count += 1 self.last_failure_time = now if self.failure_count >= self.failure_threshold: self.state = State.OPEN return fallback_func()
๐งช 6. Interactive Lab: Live Circuit Breaker & Cascading Failure Simulator
Circuit Breaker & Cascading Outage Lab
๐ฏ 7. Test Your Understanding
Your Recommendation API experiences a 10-second database lock. Within 30 seconds, the main Checkout API also crashes, even though the Checkout database is 100% healthy. Why?
Why does AWS recommend adding Full Jitter to exponential backoff algorithms during high-concurrency database recoveries?