SystemDesign.website
Track 6 ยท Module 6.1 ยท Consistency, Transactions & Distributed Consensus

CAP Theorem, PACELC & Consistency Models

Busting the "Pick 2 of 3" myth from first principles. Master CP vs AP trade-offs, the PACELC theorem, Linearizability, Causal Consistency, Eventual Consistency, and Quorum Consensus ($R + W > N$).

โฑ๏ธ Reading Time: ~28 min
๐ŸŽฏ Concepts: CAP, PACELC, Linearizability, Causal, Quorums
๐ŸŽฌ Format: Walkie-Talkie Analogy + Live 3-Node Cluster Lab

๐Ÿ“ป 1. The Core Intuition: The Walkie-Talkie Twins

๐ŸŽ’ The Analogy: Alice, Bob, and the Severed Walkie-Talkie Wire

Imagine two twin students, Alice (sitting in Classroom A) and Bob (sitting in Classroom B). They are connected by a walkie-talkie wire.

A teacher walks into Classroom A and tells Alice: "Your new score is 95". Suddenly, a student trips and cuts the walkie-talkie wire (Network Partition $P$).

A moment later, a second teacher walks into Classroom B and asks Bob: "What is the test score?". What should Bob do?

Option 1: Choose CP (Consistency over Availability)

Bob says: "My walkie-talkie wire is broken. I cannot verify if Alice received an update, so I refuse to answer right now (Error 503)".

๐Ÿ›ก๏ธ Result: Zero stale answers given (Consistency preserved), but the teacher gets an error (Availability sacrificed).
Option 2: Choose AP (Availability over Consistency)

Bob immediately answers with the old score (80) written in his notebook.

โšก Result: The system always answers immediately (Availability preserved), but the teacher gets outdated stale data (Consistency sacrificed).
๐Ÿ’ก The #1 CAP Fallacy: "Pick 2 out of 3"In real-world networks across multiple machines or data centers, Network Partitions ($P$) are a physical inevitability (cables get severed, routers reboot, GC pauses freeze threads). You cannot choose "CA without P". When a partition occurs, you MUST choose between CP or AP!

๐Ÿงฉ 2. The PACELC Theorem (What Happens When There Is No Partition?)

CAP theorem only describes what happens during a network failure. But what about the 99.9% of normal operating time? Computer scientist Daniel Abadi created PACELC:

PACELC Rule:
If there is a Partition: choose between Availability or Consistency.
Else (normal healthy operation): choose between Latency or Consistency.
DatabasePACELC TypeDuring Partition ($P$)Normal Operation ($E$)Typical Use Case
Google Spanner / CockroachDBPC / ECStops writes (Prefers C)Waits for sync consensus (Prefers C)Banking, Ledger, Financial transactions
MongoDB / HBasePC / ECMajority partition only (Prefers C)Reads from Primary (Prefers C)Document storage with strict consistency
Apache Cassandra / DynamoDBPA / ELAccepts writes anywhere (Prefers A)Returns local replica data (Prefers Low Latency)Shopping carts, Metrics, Chat messages

๐ŸŒˆ 3. The Spectrum of Distributed Consistency Models

"Consistency" is not a binary choice. It is a spectrum spanning from strongest to weakest:

1Linearizability (Strongest / Real-Time Order)

The system acts as if there is only one single copy of data in the entire universe. As soon as a write finishes at wall-clock time $t_1$, all reads anywhere on Earth at $t_2 > t_1$ MUST see the new value. (Etcd, ZooKeeper, Spanner TrueTime).

2Sequential Consistency (Leslie Lamport)

All nodes agree on the exact same chronological sequence of operations, although the sequence may lag slightly behind real-world wall-clock time.

3Causal Consistency ("Cause Precedes Effect")

If Event A causes Event B (e.g. a question on Reddit causes a reply), every user sees A before B. Independent concurrent events may be observed in different orders.

4Eventual Consistency (Weakest / Highest Throughput)

If no new updates are made, all replicas will eventually converge to the same value via Read Repair or Merkle Tree Anti-Entropy.

๐Ÿงฎ 4. Quorum Math: Guaranteeing Strong Consistency ($R + W > N$)

How do distributed databases (DynamoDB, Cassandra) provide configurable consistency without locking all nodes?

The Golden Quorum Rule:
โ€ข $N$ = Total number of replicas (e.g. $N = 3$)
โ€ข $W$ = Number of replicas that must confirm a write before returning success (e.g. $W = 2$)
โ€ข $R$ = Number of replicas that must be queried during a read (e.g. $R = 2$)
If $R + W > N$ (e.g. $2 + 2 > 3$), the Read set and Write set are guaranteed to overlap on at least 1 up-to-date node!

๐Ÿ’ป 5. Tunable Quorum Client Implementation (Multi-Language)

Here is the production implementation of a Quorum Coordinator with version resolution across 5 languages:

Select Language:
# Python Tunable Quorum Read Coordinator
import asyncio

async def quorum_read(key: str, nodes: list, R: int, N: int) -> dict:
    # Dispatch parallel reads to all N nodes
    tasks = [asyncio.create_task(node.get(key)) for node in nodes]
    responses = []
    
    for completed in asyncio.as_completed(tasks):
        resp = await completed
        if resp.is_success:
            responses.append(resp)
        if len(responses) >= R:
            break  # Quorum threshold met!

    if len(responses) < R:
        raise Exception("HTTP 503: Read Quorum Not Met")

    # Pick highest timestamp version (Last-Write-Wins / Vector Clock)
    latest = max(responses, key=lambda r: r.version)
    return latest.value

๐Ÿงช 6. Interactive Lab: Live 3-Node Network Partition Simulator

โšก Real-Time Distributed Consensus Engine

CAP Partition & Quorum Simulator

Sever transatlantic network links and test CP vs AP response!
1. System Mode:
2. Network Partition Link:
3. Write to US-East:
๐ŸŒ Node #1 (US-East)LEADER
Balance: $100
Version: v1
๐ŸŒ Node #2 (EU-Central)REPLICA
Balance: $100
Version: v1
๐ŸŒ Node #3 (AP-East)REPLICA
Balance: $100
Version: v1
๐Ÿš€ Cluster Ready (N=3, W=2, R=2): Click "โœ๏ธ Write Balance" to execute a Quorum write across all 3 global nodes!

๐ŸŽฏ 7. Test Your Understanding

Scenario 1: ATM Cash Withdrawal

You are designing an ATM withdrawal system. A user has $100 in their account. If a network partition isolates the ATM from the central bank database, should the system choose CP or AP?

Scenario 2: Social Media Comments & Replies

On a global social network, Alice posts a question: "Who won the game?", and Bob replies: "Arsenal won!". Which consistency model is strictly required to guarantee NO user sees Bob's reply before Alice's question?

โ—‹

Completed this module?

Mark as finished to update your curriculum progress tracker.