SystemDesign.website
Track 3 ยท Module 3.5 ยท DataStores & Storage Engines

Design a Distributed Key-Value Store

How systems like Amazon DynamoDB and Apache Cassandra handle millions of concurrent writes and reads with zero downtime. Master Quorum Consensus (R + W > N), Vector Clocks, Sloppy Quorums, Hinted Handoff, and Merkle Trees.

โฑ๏ธ Reading Time: ~30 min
๐ŸŽฏ Classic System: Amazon Dynamo & Apache Cassandra (Alex Xu Ch. 6)
๐ŸŽฌ Format: Voting Analogy + Step-by-Step SVGs + Live Lab

๐Ÿ“ฆ 1. Why a Single Key-Value Server Fails

A Key-Value Store is a non-relational database where data is stored as a collection of (key, value) pairs. The API is deceptively simple:

put(key, value)

Inserts or updates the value associated with the specified key.

get(key)

Retrieves the value associated with the specified key.

On a single machine (like a standalone Redis instance in RAM or LevelDB on disk), everything is trivial: lookups take microseconds. But as your system grows:

1. Memory / Disk Limits:A single machine cannot hold 100 Terabytes of data in RAM or handle 500,000 queries per second.
2. Single Point of Failure (SPOF):If that single server loses power or crashes, 100% of your customer traffic goes down instantly.
๐Ÿ’ก The Distributed Architecture Goal (Masterless Dynamo):To scale infinitely, we distribute data across hundreds of peer nodes arranged in a decentralized, masterless cluster. Every node can accept reads and writes, with no central leader to crash.

โญ• 2. Data Partitioning & Multi-Node Replication

How do we decide which server holds which key, and how do we protect against hardware crashes? We combine two core techniques:

1. Consistent Hash Ring

Both servers and keys are hashed onto a 360ยฐ circular continuum (0 to 2ยฒโตโถ-1). A key is assigned to the first node encountered moving clockwise.

2. Replication Factor (N = 3)

The first server that owns the key is the Coordinator Node. To survive crashes, the coordinator replicates the key to the next N-1 distinct physical servers clockwise on the ring.

1 Multi-Node Replica Placement (Replication Factor N = 3)

When key user_42 is written, it hashes to a position before Node A. Node A acts as the Coordinator and replicates the write to the next 2 physical nodes (Node B and Node C).

Node A (Coordinator)Node B (Replica 1)Node C (Replica 2)Node DNode Euser_42REPLICATIONN = 3 Nodes

โš–๏ธ 3. Tunable Consistency & Quorum Consensus (R + W > N)

๐Ÿ—ณ๏ธ The Simple Analogy: Committee Voting

Imagine a 5-member board (N = 5). To write a new law, you require at least 3 members to vote YES (W = 3). Later, to read the latest law, you ask any 3 random members (R = 3).

Because 3 + 3 = 6 > 5, at least ONE member in your read group is guaranteed to have voted in the write group!

Replication Factor:NTotal number of replica nodes storing the key.
Write Quorum:WNumber of replica nodes that must ACK a write before success.
Read Quorum:RNumber of replica nodes that must respond to a read query.
๐ŸŒŸ The Quorum Consensus Rule:
โ€ข If W + R > N $\implies$ Strong Consistency (Read is mathematically guaranteed to return the latest written value).
โ€ข If W + R ≤ N $\implies$ Eventual Consistency (High throughput, but reads might return stale data).
ConfigurationQuorum MathConsistency LevelIdeal Workload / Real-World Use Case
N=3, W=2, R=22 + 2 = 4 > 3Strong ConsistencyBalanced Production Default: E-commerce shopping carts, user authentication tokens.
N=3, W=1, R=31 + 3 = 4 > 3Strong ConsistencyWrite-Heavy Ingestion: High-volume IoT telemetry, fast log ingestion with slow background reporting.
N=3, W=3, R=13 + 1 = 4 > 3Strong ConsistencyRead-Heavy Fast Lookups: Product catalogs, currency exchange rates, video metadata.
N=3, W=1, R=11 + 1 = 2 ≤ 3Eventual ConsistencyMaximum Availability & Lowest Latency: View count tickers, social media feed counters.

โฑ๏ธ 4. Inconsistency Resolution with Vector Clocks

Why can't distributed databases simply use wall-clock timestamps (e.g. timestamp: 2026-08-22 14:00:00.123) to pick the winner?

โš ๏ธ The Clock Drift Problem (NTP Skew):

Physical clocks on different servers are never perfectly in sync. Server A's clock might drift 30 milliseconds ahead of Server B. If you use "Last-Write-Wins" (LWW), Server A can silently overwrite a newer write from Server B, causing silent data loss!

To solve this without wall clocks, DynamoDB introduced Vector Clocks. A Vector Clock is a list of server-version pairs:VC = [ (Server_1, Version_v1), (Server_2, Version_v2), ... ]

2 Step-by-Step: Concurrent Writes & Client-Side Reconciliation
D1: Cart = [Item A]VC: ([S1, 1])D2: Cart = [Item A, B]VC: ([S1, 2])D3: Added Item CVC: ([S1, 2], [S2, 1])D4: Added Item DVC: ([S1, 2], [S3, 1])D5: Client Merges [A, B, C, D]VC: ([S1, 3], [S2, 1], [S3, 1])Conflict Resolved with Zero Data Loss

๐Ÿ›ก๏ธ 5. Handling Failures in Distributed Clusters

In a cluster of 1,000 machines, server crashes, power supply reboots, and fiber-optic network glitches are not rare anomalies โ€” they happen every single hour. Here is how distributed KV stores survive:

1Gossip Protocol (Decentralized Heartbeats)

There is no single master node polling every server. Instead, each node randomly selects 3 peer neighbors every second and exchanges a heartbeat table. If Node A doesn't increment its heartbeat counter for 10 seconds, rumors spread across the cluster in O(log N) rounds until all nodes agree Node A is marked dead.

2Sloppy Quorum & Hinted Handoff (Temporary Glitches)

If Node A is temporarily down when a write arrives, the coordinator chooses the first healthy neighbor downstream on the ring (e.g. Node D) to store the write temporarily with a "hint" attached. When Node A comes back online, Node D hands the data back (**Hinted Handoff**), achieving 100% write availability.

3Anti-Entropy with Merkle Trees (Permanent Replica Sync)

If a replica was offline for 3 days and missed millions of writes, how do two replicas find divergent keys without sending gigabytes over the network?

They compare Merkle Trees (Hash Trees). If the root hashes match, all billions of keys are identical in O(1) time! If roots differ, they compare child hashes down the tree to isolate and synchronize only the exact divergent key range.

๐Ÿ—๏ธ 6. The Complete Write & Read Storage Engine Path

When a node receives a write or read request, what happens under the hood? Cassandra and DynamoDB use an LSM-Tree Storage Engine:

โšก The Write Path (Blazing Fast O(1))
1. Write is appended to sequential Commit Log (WAL) on disk for durability.
2. Write is inserted into in-memory sorted buffer (MemTable).
3. Write immediately returns success (0 random disk seeks!).
4. When MemTable fills up, it flushes to disk as an immutable SSTable.
๐Ÿ” The Read Path (Multi-Layer Check)
1. Check MemTable in RAM (if found, return immediately).
2. Check Bloom Filter (eliminates 99% of useless disk reads if key is absent).
3. Consult SSTable Row Index in memory to locate disk block.
4. Perform Read Repair across replicas if values differ.

๐Ÿ’ป 7. Production Quorum Coordinator Implementation

Below is the core algorithm used by the Coordinator Node to execute quorum writes with concurrent acks:

Select Language:
// Python 3.10+ Async Quorum Consensus Coordinator
import asyncio

class CoordinatorNode:
    def __init__(self, ring, replication_factor: int = 3, write_quorum: int = 2):
        self.ring = ring
        self.N = replication_factor
        self.W = write_quorum

    async def put(self, key: str, value: str) -> bool:
        # 1. Locate next N distinct replica nodes on hash ring
        replicas = self.ring.get_preference_list(key, self.N)
        
        # 2. Fire concurrent asynchronous writes to all N replicas
        tasks = [asyncio.create_task(node.write(key, value)) for node in replicas]
        
        acks = 0
        for completed_task in asyncio.as_completed(tasks, timeout=2.0):
            try:
                success = await completed_task
                if success: acks += 1
                # Quorum reached! Return success without waiting for slowest replica
                if acks >= self.W: return True
            except Exception: continue
            
        raise TimeoutError("Quorum consensus write failed to meet W threshold")

๐Ÿงช 8. Interactive Lab: Live Distributed Cluster Simulator

โšก Real-Time Cluster Canvas

Interactive Quorum & Fault Simulator

Kill nodes, adjust W/R quorums, and test data consistency
1. Write Quorum (W):
W = 2
2. Read Quorum (R):
R = 2
3. Test Operation:
Click a node to Toggle Crash / Online State:
Quorum Consensus Health:
W(2) + R(2) = 4 > N(3) โž” STRONG CONSISTENCY

Guaranteed fresh reads. At least 1 replica in the read quorum overlaps with the write quorum.

Last Operation Log:
Ready. Click "Write Key" or "Read Key" above.

๐ŸŽฏ 9. Test Your Understanding

Scenario 1: The Shopping Cart Disappearing Item Bug

Amazon's early shopping cart system used Last-Write-Wins (LWW) with NTP timestamps. Customers complained that items they added on their phone occasionally vanished when they opened their laptop. Why?

Scenario 2: The Fast Ingestion Tradeoff

A ride-sharing platform tracks live GPS locations of 2,000,000 drivers sending coordinates every 2 seconds. Which Quorum configuration provides the highest throughput with zero write bottlenecks?

โ—‹

Completed this module?

Mark as finished to update your curriculum progress tracker.