SystemDesign.website
Track 3 ยท Module 3.8 ยท DataStores & Storage Engines

Zero-Downtime Database Migrations

How Stripe, Uber, and Netflix alter schemas and migrate terabytes of data across databases without dropping a single user request. Master the Expand & Contract pattern, Dual Writing, Backfilling, and GitHub's gh-ost.

โฑ๏ธ Reading Time: ~25 min
๐ŸŽฏ Production Tools: Expand & Contract, gh-ost, CDC (Debezium)
๐ŸŽฌ Format: Bridge Analogy + Step-by-Step SVGs + Live Lab

๐Ÿ’ฅ 1. Why Naive Schema Changes Crash Production

Imagine you run a fast-growing application with 50,000,000 users. You need to split a name column into first_name and last_name.

A junior engineer suggests running this simple SQL migration:

-- โŒ THE DISASTER MIGRATION SCRIPT
ALTER TABLE users ADD COLUMN first_name VARCHAR(100);
ALTER TABLE users ADD COLUMN last_name VARCHAR(100);
UPDATE users SET first_name = SPLIT_STR(name, ' ', 1);
ALTER TABLE users DROP COLUMN name;
โš ๏ธ What actually happens in production:
1. Table Lock: The ALTER TABLE statement acquires an exclusive metadata lock on the table.
2. Traffic Freeze: All 10,000 incoming user requests per second are queued up and blocked.
3. Connection Pool Exhaustion: The application servers run out of database connections within 3 seconds.
4. Global Outage: The entire website goes down with 504 Gateway Timeouts for 45 minutes until the table finishes copying!
๐Ÿ’ก The 24/7 Production Rule:"Scheduled Maintenance Windows" are dead. Modern global platforms must perform database schema changes and engine migrations while users continue buying products, sending messages, and updating profiles with ZERO downtime!

๐ŸŒ‰ 2. The Golden Solution: The Expand & Contract Pattern

๐ŸŒ‰ The Simple Analogy: Building a New Bridge Next to the Old One

How do engineers replace a massive highway bridge without shutting down traffic?

They don't demolish the old bridge first. Instead, they build a brand-new bridge right beside the old one. They let traffic flow across both bridges in parallel, verify safety, reroute cars to the new bridge, and only then dismantle the old bridge.

In software engineering, this is called the Expand & Contract (Parallel Run) Pattern:

1Phase 1: Expand (Add New Columns/Tables)

Add the new columns (first_name, last_name) with a default or nullable value. Old application code continues writing and reading from name.

2Phase 2: Dual-Writing (Write to BOTH Old and New)

Deploy new application code that writes incoming user updates to BOTH the old name and new (first_name, last_name). Reads still come from the old column.

3Phase 3: Backfilling Historical Data (Non-Blocking Batches)

Run a background migration worker that slowly reads historical rows where first_name IS NULL in small chunks of 1,000 rows with sleep delays, populating the new columns without stressing the database.

4Phase 4: Shadow Reads & Verification

The app reads from both Old and New columns in the background. If a mismatch is detected, an alert fires. Once 100% parity is verified, switch the main read path to the New columns via a Feature Flag.

5Phase 5: Contract (Clean Up & Drop Old Column)

Remove the dual-write logic from the application. Safely drop the old name column. The migration is complete with 0 downtime!

๐Ÿ› ๏ธ 3. Production Online Schema Tools: GitHub's gh-ost

At companies like GitHub and Shopify, human engineers don't manually write dual-writing code for every single schema change. Instead, they use Online Schema Change (OSC) engines like gh-ost:

1. The Ghost Table Trick

gh-ost creates a temporary ghost table (e.g. _users_gho) with the desired new columns and indexes.

2. Binary Log (binlog) Streaming

Instead of using heavy database triggers (which slow down production queries), gh-ost tails the MySQL binlog stream to replicate live updates to the ghost table in real time.

โšก The 10-Millisecond Atomic Cutover:Once all historical data is copied and the ghost table is 100% caught up with live traffic, gh-ost executes a single atomic rename:RENAME TABLE users TO _users_old, _users_gho TO users;This swap executes in under 10 milliseconds, invisible to users!

๐Ÿ’ป 4. The Dual-Write & Feature Flag Cutover Code

Here is the exact application-level Dual-Write and Feature Flag logic used in production:

Select Language:
// Python 3.10+ Dual-Write & Feature Flag Cutover
class UserRepository:
    def __init__(self, db, feature_flags):
        self.db = db
        self.flags = feature_flags

    def update_user_name(self, user_id: int, full_name: str):
        first_name, last_name = full_name.split(" ", 1)

        # Phase 2 & 3: Dual Write (Write to both Old and New columns)
        sql = "UPDATE users SET name = %s, first_name = %s, last_name = %s WHERE id = %s"
        self.db.execute(sql, (full_name, first_name, last_name, user_id))

    def get_user_name(self, user_id: int) -> str:
        # Phase 4 & 5: Controlled Read Cutover via Feature Flag
        if self.flags.is_enabled("read_new_name_schema"):
            row = self.db.query("SELECT first_name, last_name FROM users WHERE id = %s", (user_id,))
            return f"{row.first_name} {row.last_name}"
        else:
            row = self.db.query("SELECT name FROM users WHERE id = %s", (user_id,))
            return row.name

๐Ÿงช 5. Interactive Lab: Live 5-Phase Zero-Downtime Migration

โšก Real-Time Migration Pipeline

Interactive 5-Phase Cutover Simulator

Advance through the 5 migration phases with live incoming traffic!
Database Table: usersPhase 1: Old Schema
Background Backfiller0% Completed

Copies historical data in batches of 1,000 rows without locking tables.

๐Ÿš€ Phase 1 Active: Application is running on the old schema. Click "Advance to Phase 2 (Dual Write)" above to begin!

๐ŸŽฏ 6. Test Your Understanding

Scenario 1: The Race Condition Disaster

An engineer backfills historical data BEFORE enabling Dual-Writing in the application code. Why does this cause permanent data corruption?

Scenario 2: Instant Rollback Protection

During Phase 4 (Shadow Reads), an unexpected bug is found in the new schema parser. How does the Expand & Contract pattern allow an instant 0-second rollback with zero data loss?

โ—‹

Completed this module?

Mark as finished to update your curriculum progress tracker.