SystemDesign.website
Track 4 ยท Module 4.3 ยท Caching & Memory Architecture

Bloom Filters & Probabilistic Data Structures

How Google, Instagram, and Cassandra test membership for billions of items using 99% less memory. Master Bloom Filters, HyperLogLog, and Count-Min Sketch from mathematical first principles.

โฑ๏ธ Reading Time: ~26 min
๐ŸŽฏ Mathematical Core: Bit Arrays, Hash Functions, False Positive Rates
๐ŸŽฌ Format: Club Bouncer Analogy + Interactive Bit Array Lab

๐Ÿง  1. The Problem: When Exact Sets Run Out of RAM

๐ŸŽŸ๏ธ The Analogy: The Nightclub Bouncer's Hand Stamp

Imagine a popular nightclub with 10,000 guests.

If the bouncer has to look up every returning guest's full name, photo, and ID in a massive 500-page book (a standard Hash Set), entry takes forever and the book takes up an entire desk.

Instead, the bouncer stamps each guest's wrist with a special glowing ultraviolet stamp. When someone wants to re-enter, a 0.1-second glance under the UV light tells the bouncer:"No stamp? You were DEFINITELY NOT inside. Stamp present? You were ALMOST CERTAINLY inside!"

In software architecture, this is the power of Probabilistic Data Structures:

โŒ Exact Hash Set (1 Billion Users)

Storing 1 Billion 32-byte strings in an exact hash set requires over 32 Gigabytes of expensive RAM.

Accuracy: 100% | RAM Cost: $350 / month
โœ… Bloom Filter (1 Billion Users)

A Bloom filter can test membership of 1 Billion items with a 1% error rate in just 1.2 Gigabytes of RAM (96% less memory!).

Accuracy: 99% | RAM Cost: $12 / month
๐Ÿ’ก The Golden Law of Probabilistic Structures:No means definitely NO (0% False Negatives). If a Bloom Filter says an item is not in the set, it is 100% guaranteed not to exist.
Yes means PROBABLY yes (Small False Positive chance). If it says yes, it might just be hash collision noise, which we can verify against disk.

๐Ÿ”ฌ 2. How a Bloom Filter Works (Step-by-Step)

A Bloom Filter consists of two simple mathematical building blocks:
1. A Bit Array of size m bits, initialized to all 0s.
2. k independent, fast hash functions (e.g. h1, h2, h3).

1Step 1: Adding an Item (e.g. "alice")

Pass "alice" through all 3 hash functions: h1("alice") = 2, h2("alice") = 7, h3("alice") = 11. Set bits 2, 7, and 11 to 1.

2Step 2: Adding Another Item (e.g. "bob")

Pass "bob" through the hash functions: h1("bob") = 5, h2("bob") = 7, h3("bob") = 14. Set bits 5, 7, and 14 to 1 (Note: bit 7 was already 1, which is totally fine!).

3Step 3: Querying "charlie" (Definite Negative)

Check hashes: h1("charlie") = 2 (1), h2("charlie") = 3 (0). Because bit 3 is 0, "charlie" was DEFINITELY NEVER ADDED! We can return 404 immediately without touching the database disk!

4Step 4: The False Positive Case

Querying "eve" hashes to bits 2, 5, 11. All three bits are 1 (because "alice" set 2 and 11, and "bob" set 5!). The filter says "probably present", triggering a DB lookup. This is a harmless False Positive.

๐Ÿ“Š 3. HyperLogLog & Count-Min Sketch: Counting at Billions Scale

Bloom filters test set membership. What if you need to count unique visitors or find top trending items?

1. HyperLogLog (Count Unique Users)

Estimates the cardinality (number of unique visitors / DAU) of millions of users using only 12 Kilobytes of RAM with an error rate of just 0.81%.

๐Ÿช™ The Coin Analogy: If you flip coins and observe a streak of 10 consecutive Tails, you can mathematically infer you flipped roughly 210 = 1,024 times. Redis command: PFADD & PFCOUNT.
2. Count-Min Sketch (Heavy Hitters)

Tracks the frequency of events in high-volume live streams (e.g. top trending hashtags on Twitter or top DDoS attacker IPs) in fixed sub-megabyte memory.

๐Ÿ“ˆ The Mechanism: Uses a 2D matrix of counters. Multiple hash functions increment row counters on writes, and reads take the MIN() across all rows to minimize overestimation.

๐Ÿ’ป 4. Production Bloom Filter Implementation (Multi-Language)

Below is the complete implementation of a Bloom Filter with multiple hash functions across 5 programming languages:

Select Language:
// Python 3.10+ Bloom Filter
import hashlib

class BloomFilter:
    def __init__(self, size_in_bits: int, hash_count: int):
        self.size = size_in_bits
        self.k = hash_count
        self.bit_array = bytearray((size_in_bits + 7) // 8)

    def _hashes(self, item: str):
        # Kirsch-Mitzenmacher optimization: generate k hashes from 2 hash values
        h1 = int(hashlib.sha256(item.encode()).hexdigest(), 16)
        h2 = int(hashlib.md5(item.encode()).hexdigest(), 16)
        for i in range(self.k):
            yield (h1 + i * h2) % self.size

    def add(self, item: str):
        for bit in self._hashes(item):
            self.bit_array[bit // 8] |= (1 << (bit % 8))

    def contains(self, item: str) -> bool:
        for bit in self._hashes(item):
            if not (self.bit_array[bit // 8] & (1 << (bit % 8))):
                return False  # Definitely not present (0% false negative)
        return True  # Probably present

๐Ÿงช 5. Interactive Lab: Live 16-Bit Bloom Filter

โšก Real-Time Bit Array Visualizer

Interactive 16-Bit Array & 3 Hash Functions

Insert keys and watch hash beams flip bits in real-time!
Bit Array (m = 16 bits, k = 3 Hashes)0 / 16 Bits Set (0%)
๐Ÿš€ Ready: Enter a key name and click "โž• Add Key" to see bits flip from 0 to 1!

๐ŸŽฏ 6. Test Your Understanding

Scenario 1: Why Can't You Delete from a Standard Bloom Filter?

A developer tries to delete a user by setting the hash bit indices back to 0. Why is this forbidden in a standard Bloom Filter?

Scenario 2: The YouTube Live View Counter

You need to calculate the exact unique number of monthly active users (500 Million users) across 10,000 live channels with minimal RAM. Which data structure should you choose?

โ—‹

Completed this module?

Mark as finished to update your curriculum progress tracker.