Service Discovery & Health Checks
How microservices find each other in dynamic cloud environments where containers autoscale, crash, and restart every minute. Master Client-Side vs Server-Side Discovery, Heartbeat Leases, Gossip Protocols (SWIM / Serf), and Envoy Outlier Detection.
โ๏ธ 1. The Core Intuition: The Airport Departures & Gate Board
Imagine an international airport with 100 arrival and departure gates:
Planes dock, refuel, depart, and get reassigned to different gates constantly.
Passengers don't memorize static gate numbers. Instead, everyone looks at the central Airport Departures Board (The Service Registry), which displays the live gate location of every flight in real-time.
Every pilot radios Air Traffic Control every few minutes (Heartbeats). If a flight goes completely silent for 15 minutes, it is automatically removed from the active departures board (Deregistration & Eviction).
1. Autoscaling launches 30 new pods during flash-sales, assigning random private IPs (e.g.
10.244.2.19).2. Pods crash, restart on different worker nodes, and change IP addresses in seconds.
3. Hardcoded IPs or static DNS with long TTLs cause clients to send traffic to dead containers, causing massive
HTTP 502/504 Bad Gateway outages!โ๏ธ 2. Client-Side vs Server-Side Discovery Architecture
There are two fundamental ways microservices locate each other:
The calling client queries the Service Registry directly, caches the list of available IP addresses locally in memory, and performs its own client-side load balancing (Round Robin / Least Conns).
The client calls a standard virtual DNS name (http://order-service). A smart router (kube-proxy, AWS ALB, or Envoy Sidecar) looks up the registry and forwards the packet.
๐ 3. Heartbeat Leases, Eviction & Eureka Self-Preservation
How does a registry know when a container dies without manually polling every endpoint?
Upon booting, the service registers its IP with a TTL lease (e.g. 90 seconds). A background thread fires a PUT /apps/payment-service/{instanceId} heartbeat every 30 seconds to renew the lease.
If a container crashes or gets killed by Kubernetes OOMKiller, heartbeats stop. After 3 missed heartbeats (90 seconds), the registry removes the instance from the routing table!
If a sudden network partition disconnects 60% of instances at once, Eureka recognizes that the instances might still be healthy and the network is at fault. It enters Self-Preservation Mode, pausing all evictions to prevent destroying the entire cluster!
๐ฃ๏ธ 4. Decentralized Gossip Protocols (SWIM / HashiCorp Serf)
In large clusters with 50,000+ nodes, a centralized registry becomes a bottleneck. How do databases like Cassandra and Consul discover nodes without a master?
Every node periodically pings a random peer. If the peer doesn't reply, it asks 3 other nodes to ping it. If all fail, the node is marked as dead and gossiped to the entire cluster.
โก Scalability: Scales logarithmically ($O(\log N)$ network messages) with zero single point of failure!Rather than sending active health checks, Envoy monitors real user requests. If a pod returns 5 consecutive HTTP 500 errors, Envoy temporarily ejects it from the pool for 30 seconds.
๐ป 5. Production Service Discovery Client (Multi-Language)
Here is the production implementation of a dynamic Service Discovery client with background heartbeat renewal across 5 languages:
# Python Service Registration & Background Heartbeat Loop import asyncio import httpx class ConsulClient: def __init__(self, service_name: str, ip: str, port: int, consul_url: str): self.service_name = service_name self.instance_id = f"{service_name}-{ip}:{port}" self.ip = ip self.port = port self.consul_url = consul_url self.is_running = True async def register(self): payload = { "ID": self.instance_id, "Name": self.service_name, "Address": self.ip, "Port": self.port, "Check": {"TTL": "30s", "DeregisterCriticalServiceAfter": "90s"} } async with httpx.AsyncClient() as client: await client.put(f"{self.consul_url}/v1/agent/service/register", json=payload) asyncio.create_task(self.start_heartbeat()) async def start_heartbeat(self): async with httpx.AsyncClient() as client: while self.is_running: await client.put(f"{self.consul_url}/v1/agent/check/pass/service:{self.instance_id}") await asyncio.sleep(10) # Heartbeat every 10 seconds
๐งช 6. Interactive Lab: Live Dynamic Cluster & Heartbeat Simulator
Dynamic Service Registry & Heartbeat Dashboard
payment-service registered. Click "Route 1 Request" to test client-side load balancing!๐ฏ 7. Test Your Understanding
During a rolling deployment in Kubernetes, your payment pods get new IP addresses. However, upstream Java microservices keep hitting the old terminated pod IPs for 10 minutes. What is the root cause?
A network switch failure causes 70% of microservice instances to lose connection to the Netflix Eureka server. Why does Eureka intentionally enter Self-Preservation Mode instead of evicting all 70% of instances?