SystemDesign.website
Track 7 ยท Module 7.3 ยท Distributed Reliability, Rate Limiting & DevOps

Service Discovery & Health Checks

How microservices find each other in dynamic cloud environments where containers autoscale, crash, and restart every minute. Master Client-Side vs Server-Side Discovery, Heartbeat Leases, Gossip Protocols (SWIM / Serf), and Envoy Outlier Detection.

โฑ๏ธ Reading Time: ~28 min
๐ŸŽฏ Architecture: Eureka, Consul, Kubernetes CoreDNS, Gossip Protocol
๐ŸŽฌ Format: Airport Departures Analogy + Live Dynamic Registry Lab

โœˆ๏ธ 1. The Core Intuition: The Airport Departures & Gate Board

๐Ÿ›ซ The Analogy: The Dynamic Flight Gate Board

Imagine an international airport with 100 arrival and departure gates:
Planes dock, refuel, depart, and get reassigned to different gates constantly.

Passengers don't memorize static gate numbers. Instead, everyone looks at the central Airport Departures Board (The Service Registry), which displays the live gate location of every flight in real-time.

Every pilot radios Air Traffic Control every few minutes (Heartbeats). If a flight goes completely silent for 15 minutes, it is automatically removed from the active departures board (Deregistration & Eviction).

๐Ÿ’ฅ Why Hardcoded IP Addresses Fail in Modern CloudsIn Kubernetes and AWS ECS, containers are ephemeral:
1. Autoscaling launches 30 new pods during flash-sales, assigning random private IPs (e.g. 10.244.2.19).
2. Pods crash, restart on different worker nodes, and change IP addresses in seconds.
3. Hardcoded IPs or static DNS with long TTLs cause clients to send traffic to dead containers, causing massive HTTP 502/504 Bad Gateway outages!

โš–๏ธ 2. Client-Side vs Server-Side Discovery Architecture

There are two fundamental ways microservices locate each other:

1. Client-Side Discovery (Netflix Eureka / Consul)

The calling client queries the Service Registry directly, caches the list of available IP addresses locally in memory, and performs its own client-side load balancing (Round Robin / Least Conns).

โ€ข Pros: Direct point-to-point connection (fastest latency, zero middleman proxy).
โ€ข Cons: Couples client code to registry SDKs in every programming language.
2. Server-Side Discovery (Kubernetes DNS + kube-proxy / Envoy)

The client calls a standard virtual DNS name (http://order-service). A smart router (kube-proxy, AWS ALB, or Envoy Sidecar) looks up the registry and forwards the packet.

โ€ข Pros: 100% language-agnostic; client code needs zero discovery libraries.
โ€ข Cons: Extra network hop through the proxy router.

๐Ÿ’“ 3. Heartbeat Leases, Eviction & Eureka Self-Preservation

How does a registry know when a container dies without manually polling every endpoint?

1Heartbeat Renewal (Every 30 Seconds)

Upon booting, the service registers its IP with a TTL lease (e.g. 90 seconds). A background thread fires a PUT /apps/payment-service/{instanceId} heartbeat every 30 seconds to renew the lease.

2Eviction on Lease Expiry (90 Seconds)

If a container crashes or gets killed by Kubernetes OOMKiller, heartbeats stop. After 3 missed heartbeats (90 seconds), the registry removes the instance from the routing table!

3Eureka Self-Preservation Mode (Network Split Protection)

If a sudden network partition disconnects 60% of instances at once, Eureka recognizes that the instances might still be healthy and the network is at fault. It enters Self-Preservation Mode, pausing all evictions to prevent destroying the entire cluster!

๐Ÿ—ฃ๏ธ 4. Decentralized Gossip Protocols (SWIM / HashiCorp Serf)

In large clusters with 50,000+ nodes, a centralized registry becomes a bottleneck. How do databases like Cassandra and Consul discover nodes without a master?

SWIM Gossip Protocol

Every node periodically pings a random peer. If the peer doesn't reply, it asks 3 other nodes to ping it. If all fail, the node is marked as dead and gossiped to the entire cluster.

โšก Scalability: Scales logarithmically ($O(\log N)$ network messages) with zero single point of failure!
Passive Outlier Detection (Envoy)

Rather than sending active health checks, Envoy monitors real user requests. If a pod returns 5 consecutive HTTP 500 errors, Envoy temporarily ejects it from the pool for 30 seconds.

๐Ÿ’ป 5. Production Service Discovery Client (Multi-Language)

Here is the production implementation of a dynamic Service Discovery client with background heartbeat renewal across 5 languages:

Select Language:
# Python Service Registration & Background Heartbeat Loop
import asyncio
import httpx

class ConsulClient:
    def __init__(self, service_name: str, ip: str, port: int, consul_url: str):
        self.service_name = service_name
        self.instance_id = f"{service_name}-{ip}:{port}"
        self.ip = ip
        self.port = port
        self.consul_url = consul_url
        self.is_running = True

    async def register(self):
        payload = {
            "ID": self.instance_id,
            "Name": self.service_name,
            "Address": self.ip,
            "Port": self.port,
            "Check": {"TTL": "30s", "DeregisterCriticalServiceAfter": "90s"}
        }
        async with httpx.AsyncClient() as client:
            await client.put(f"{self.consul_url}/v1/agent/service/register", json=payload)
        asyncio.create_task(self.start_heartbeat())

    async def start_heartbeat(self):
        async with httpx.AsyncClient() as client:
            while self.is_running:
                await client.put(f"{self.consul_url}/v1/agent/check/pass/service:{self.instance_id}")
                await asyncio.sleep(10)  # Heartbeat every 10 seconds

๐Ÿงช 6. Interactive Lab: Live Dynamic Cluster & Heartbeat Simulator

โšก Real-Time Service Registry Engine

Dynamic Service Registry & Heartbeat Dashboard

Spawn pods, simulate missed heartbeats, and watch automated evictions!
1. Cluster Scaling:
2. Simulate Outage:
3. Client Traffic:
PaymentService Live Registry Pool3 Active Pods
๐Ÿš€ Service Registry Active: 3 instances of payment-service registered. Click "Route 1 Request" to test client-side load balancing!

๐ŸŽฏ 7. Test Your Understanding

Scenario 1: Stale DNS Caching in Kubernetes

During a rolling deployment in Kubernetes, your payment pods get new IP addresses. However, upstream Java microservices keep hitting the old terminated pod IPs for 10 minutes. What is the root cause?

Scenario 2: Eureka Self-Preservation Mode

A network switch failure causes 70% of microservice instances to lose connection to the Netflix Eureka server. Why does Eureka intentionally enter Self-Preservation Mode instead of evicting all 70% of instances?

โ—‹

Completed this module?

Mark as finished to update your curriculum progress tracker.