Database Indexes & B-Tree Internals
How database storage engines locate a single row out of 100 million in under 2 milliseconds. Master disk page block I/O, B+ Tree fan-out mechanics, Clustered vs Secondary indexes, Covering Indexes, and the severe write-penalty trade-off.
💥 1. The Real-World Nightmare: The Full Table Scan
Imagine you have an 800-page computer science textbook without an index in the back. You need to find the definition of "Consistent Hashing". What must you do?
You have no choice but to start at Page 1 and scan every single line of text on every single page until you reach Page 742. This is an O(N) Full Table Scan.
SELECT * FROM users WHERE email = 'alice@gmail.com';Without an index on email, the storage engine must load all 125,000 pages (2 GB) from disk into RAM sequentially. Even at 150 MB/s disk speed, this single query takes 13.3 SECONDS, saturates 100% of Disk I/O, and causes the entire database server to freeze.📑 2. What is an Index, Really?
An Index is an auxiliary, sorted data structure kept alongside the table. It holds only two items:
The value you are searching for (e.g. alice@gmail.com), kept in strict sorted alphabetical order.
The exact physical location of the full row on disk (e.g. Page #412, Slot #14 or Primary Key id = 8941).
A table can only be physically stored in ONE sorted order on disk (e.g. by id). But in production, you need fast lookups by email, username, phone_number, and created_at. Each auxiliary index gives you a fast-path lookup for that specific column.
🌳 3. Why Binary Search Trees Fail on Disk (The Birth of B-Trees)
In university algorithms courses, we learn that balanced Binary Search Trees (Red-Black Trees, AVL Trees) give fast O(log₂ N) lookups. Why don't production databases like MySQL, Postgres, Oracle, or SQLite use Binary Search Trees?
log₂(10,000,000) ≈ 24 levels.log₅₀₀(10,000,000) ≈ 3 levels.Every level you descend in a disk tree requires a separate disk page fetch. Increasing the fan-out from 2 to 500 flattens the tree into a broad, shallow 3-level structure.
⚡ 4. B-Tree vs B+ Tree: The 3 Rules of Modern Databases
Almost every relational and key-value database in production (MySQL InnoDB, Postgres, SQLite, Oracle) uses a B+ Tree rather than a standard B-Tree. Here are the 3 structural rules that make B+ Trees supreme:
In a standard B-Tree, internal nodes store actual table rows. In a B+ Tree, internal nodes store strictly search keys and child page pointers (e.g. [Key: 50 | Pointer: Page 14]). Because keys are tiny (8 bytes), a single 16 KB page holds 1,000+ keys, maximizing fan-out and keeping the tree razor-flat.
Every query traverses to the bottom leaf level. Every point lookup takes the exact same predictable number of I/O steps (O(log N)).
All leaf pages are linked horizontally left-to-right via pointers (prev <--> next). This makes Range Scans lightning fast!
SELECT * FROM orders WHERE amount BETWEEN 100 AND 500: the engine does 1 single tree lookup to find leaf node 100, then simply walks right along the linked list pages without ever backtracking up the tree!🔍 5. Clustered vs Secondary Index & The Double Lookup
Understanding the difference between Clustered and Secondary (Non-Clustered) indexes is the #1 question asked in senior system design and database interviews.
In MySQL InnoDB, the table IS the clustered index. The leaf pages do not store pointers — they store the actual, complete row columns (id, name, email, password, created_at).
An auxiliary B+ Tree created on a non-primary column. The leaf pages store the indexed value plus the Primary Key value (e.g. email -> id: 8941).
When you run SELECT * FROM users WHERE email = 'alice@gmail.com':
email $\implies$ finds leaf containing id = 8941 (3 page reads).id = 8941 and traverses the Clustered Index B+ Tree $\implies$ retrieves the full table row (3 page reads).If you write: SELECT id, email FROM users WHERE email = 'alice@gmail.com', the database finds both email and id directly inside the secondary index leaf node. It skips Step 2 completely! This is called an Index-Only Scan / Covering Index (0 clustered disk seeks).
⚖️ 6. The Hidden Cost: The Write Penalty & Page Splits
Why not just index every column in the table? Because indexes speed up READS, but drastically slow down WRITES.
When you insert 1 row into a table with 6 secondary indexes, the database must write to the table AND perform 6 separate B+ Tree rebalancing writes on disk!
When a 16 KB leaf page fills up, inserting a new key forces the storage engine to split the page into two 8 KB pages, allocate new disk space, and update parent tree pointers up to the root.
| Number of Indexes on Table | SELECT Latency (Point Query) | INSERT / Batch Write Throughput | Recommendation |
|---|---|---|---|
| 0 Indexes (Heap Scan) | 13,000 ms (Catastrophic) | 65,000 writes/sec | Terrible for queries |
| 1 – 3 Indexes (Targeted) | 1.2 ms (Optimal) | 48,000 writes/sec | Production Sweet Spot |
| 8+ Indexes (Over-Indexed) | 1.2 ms | 6,500 writes/sec (10x drop!) | Write bottleneck & RAM bloat |
💻 7. Low-Level B+ Tree Node Search Implementation
How does a database storage engine search inside a 16 KB page? It uses binary search over the sorted page keys in O(log K) time.
// Python 3.10+ B+ Tree Node Traversal & Binary Search import bisect class BPlusTreeNode: def __init__(self, is_leaf: bool = False): self.is_leaf = is_leaf self.keys = [] # Sorted keys in 16KB page self.children = [] # Child page pointers (if internal) self.records = [] # Row data / RecordIDs (if leaf) self.next = None # Horizontal linked list pointer def find_child(self, key): # O(log K) binary search to find child page index idx = bisect.bisect_right(self.keys, key) return self.children[idx] def search(self, key): curr = self # Descend from Root to Leaf while not curr.is_leaf: curr = curr.find_child(key) # Search within leaf page idx = bisect.bisect_left(curr.keys, key) if idx < len(curr.keys) and curr.keys[idx] == key: return curr.records[idx] return None
🧪 8. Interactive Lab: Live B+ Tree & Query Execution Plan
Interactive B+ Tree Search & EXPLAIN Simulator
SELECT * FROM users WHERE id = 30;🎯 9. Test Your Understanding
A backend engineer adds 12 secondary indexes to an orders table to speed up all possible reporting queries. Later that night, a nightly batch job importing 5,000,000 orders takes 14 hours instead of 15 minutes. What is the root cause?
You create an index on column is_active (BOOLEAN) in a 20,000,000 row table. When you run EXPLAIN SELECT * FROM users WHERE is_active = true, the optimizer chooses a Full Table Scan (type: ALL) instead of using your index. Why?