Zero-Downtime Database Migrations
How Stripe, Uber, and Netflix alter schemas and migrate terabytes of data across databases without dropping a single user request. Master the Expand & Contract pattern, Dual Writing, Backfilling, and GitHub's gh-ost.
๐ฅ 1. Why Naive Schema Changes Crash Production
Imagine you run a fast-growing application with 50,000,000 users. You need to split a name column into first_name and last_name.
A junior engineer suggests running this simple SQL migration:
ALTER TABLE users ADD COLUMN first_name VARCHAR(100);
ALTER TABLE users ADD COLUMN last_name VARCHAR(100);
UPDATE users SET first_name = SPLIT_STR(name, ' ', 1);
ALTER TABLE users DROP COLUMN name;
ALTER TABLE statement acquires an exclusive metadata lock on the table.๐ 2. The Golden Solution: The Expand & Contract Pattern
How do engineers replace a massive highway bridge without shutting down traffic?
They don't demolish the old bridge first. Instead, they build a brand-new bridge right beside the old one. They let traffic flow across both bridges in parallel, verify safety, reroute cars to the new bridge, and only then dismantle the old bridge.
In software engineering, this is called the Expand & Contract (Parallel Run) Pattern:
Add the new columns (first_name, last_name) with a default or nullable value. Old application code continues writing and reading from name.
Deploy new application code that writes incoming user updates to BOTH the old name and new (first_name, last_name). Reads still come from the old column.
Run a background migration worker that slowly reads historical rows where first_name IS NULL in small chunks of 1,000 rows with sleep delays, populating the new columns without stressing the database.
The app reads from both Old and New columns in the background. If a mismatch is detected, an alert fires. Once 100% parity is verified, switch the main read path to the New columns via a Feature Flag.
Remove the dual-write logic from the application. Safely drop the old name column. The migration is complete with 0 downtime!
๐ ๏ธ 3. Production Online Schema Tools: GitHub's gh-ost
At companies like GitHub and Shopify, human engineers don't manually write dual-writing code for every single schema change. Instead, they use Online Schema Change (OSC) engines like gh-ost:
gh-ost creates a temporary ghost table (e.g. _users_gho) with the desired new columns and indexes.
Instead of using heavy database triggers (which slow down production queries), gh-ost tails the MySQL binlog stream to replicate live updates to the ghost table in real time.
gh-ost executes a single atomic rename:RENAME TABLE users TO _users_old, _users_gho TO users;This swap executes in under 10 milliseconds, invisible to users!๐ป 4. The Dual-Write & Feature Flag Cutover Code
Here is the exact application-level Dual-Write and Feature Flag logic used in production:
// Python 3.10+ Dual-Write & Feature Flag Cutover class UserRepository: def __init__(self, db, feature_flags): self.db = db self.flags = feature_flags def update_user_name(self, user_id: int, full_name: str): first_name, last_name = full_name.split(" ", 1) # Phase 2 & 3: Dual Write (Write to both Old and New columns) sql = "UPDATE users SET name = %s, first_name = %s, last_name = %s WHERE id = %s" self.db.execute(sql, (full_name, first_name, last_name, user_id)) def get_user_name(self, user_id: int) -> str: # Phase 4 & 5: Controlled Read Cutover via Feature Flag if self.flags.is_enabled("read_new_name_schema"): row = self.db.query("SELECT first_name, last_name FROM users WHERE id = %s", (user_id,)) return f"{row.first_name} {row.last_name}" else: row = self.db.query("SELECT name FROM users WHERE id = %s", (user_id,)) return row.name
๐งช 5. Interactive Lab: Live 5-Phase Zero-Downtime Migration
Interactive 5-Phase Cutover Simulator
Copies historical data in batches of 1,000 rows without locking tables.
๐ฏ 6. Test Your Understanding
An engineer backfills historical data BEFORE enabling Dual-Writing in the application code. Why does this cause permanent data corruption?
During Phase 4 (Shadow Reads), an unexpected bug is found in the new schema parser. How does the Expand & Contract pattern allow an instant 0-second rollback with zero data loss?