Bloom Filters & Probabilistic Data Structures
How Google, Instagram, and Cassandra test membership for billions of items using 99% less memory. Master Bloom Filters, HyperLogLog, and Count-Min Sketch from mathematical first principles.
๐ง 1. The Problem: When Exact Sets Run Out of RAM
Imagine a popular nightclub with 10,000 guests.
If the bouncer has to look up every returning guest's full name, photo, and ID in a massive 500-page book (a standard Hash Set), entry takes forever and the book takes up an entire desk.
Instead, the bouncer stamps each guest's wrist with a special glowing ultraviolet stamp. When someone wants to re-enter, a 0.1-second glance under the UV light tells the bouncer:"No stamp? You were DEFINITELY NOT inside. Stamp present? You were ALMOST CERTAINLY inside!"
In software architecture, this is the power of Probabilistic Data Structures:
Storing 1 Billion 32-byte strings in an exact hash set requires over 32 Gigabytes of expensive RAM.
Accuracy: 100% | RAM Cost: $350 / monthA Bloom filter can test membership of 1 Billion items with a 1% error rate in just 1.2 Gigabytes of RAM (96% less memory!).
Accuracy: 99% | RAM Cost: $12 / monthYes means PROBABLY yes (Small False Positive chance). If it says yes, it might just be hash collision noise, which we can verify against disk.
๐ฌ 2. How a Bloom Filter Works (Step-by-Step)
A Bloom Filter consists of two simple mathematical building blocks:
1. A Bit Array of size m bits, initialized to all 0s.
2. k independent, fast hash functions (e.g. h1, h2, h3).
Pass "alice" through all 3 hash functions: h1("alice") = 2, h2("alice") = 7, h3("alice") = 11. Set bits 2, 7, and 11 to 1.
Pass "bob" through the hash functions: h1("bob") = 5, h2("bob") = 7, h3("bob") = 14. Set bits 5, 7, and 14 to 1 (Note: bit 7 was already 1, which is totally fine!).
Check hashes: h1("charlie") = 2 (1), h2("charlie") = 3 (0). Because bit 3 is 0, "charlie" was DEFINITELY NEVER ADDED! We can return 404 immediately without touching the database disk!
Querying "eve" hashes to bits 2, 5, 11. All three bits are 1 (because "alice" set 2 and 11, and "bob" set 5!). The filter says "probably present", triggering a DB lookup. This is a harmless False Positive.
๐ 3. HyperLogLog & Count-Min Sketch: Counting at Billions Scale
Bloom filters test set membership. What if you need to count unique visitors or find top trending items?
Estimates the cardinality (number of unique visitors / DAU) of millions of users using only 12 Kilobytes of RAM with an error rate of just 0.81%.
PFADD & PFCOUNT.Tracks the frequency of events in high-volume live streams (e.g. top trending hashtags on Twitter or top DDoS attacker IPs) in fixed sub-megabyte memory.
MIN() across all rows to minimize overestimation.๐ป 4. Production Bloom Filter Implementation (Multi-Language)
Below is the complete implementation of a Bloom Filter with multiple hash functions across 5 programming languages:
// Python 3.10+ Bloom Filter import hashlib class BloomFilter: def __init__(self, size_in_bits: int, hash_count: int): self.size = size_in_bits self.k = hash_count self.bit_array = bytearray((size_in_bits + 7) // 8) def _hashes(self, item: str): # Kirsch-Mitzenmacher optimization: generate k hashes from 2 hash values h1 = int(hashlib.sha256(item.encode()).hexdigest(), 16) h2 = int(hashlib.md5(item.encode()).hexdigest(), 16) for i in range(self.k): yield (h1 + i * h2) % self.size def add(self, item: str): for bit in self._hashes(item): self.bit_array[bit // 8] |= (1 << (bit % 8)) def contains(self, item: str) -> bool: for bit in self._hashes(item): if not (self.bit_array[bit // 8] & (1 << (bit % 8))): return False # Definitely not present (0% false negative) return True # Probably present
๐งช 5. Interactive Lab: Live 16-Bit Bloom Filter
Interactive 16-Bit Array & 3 Hash Functions
๐ฏ 6. Test Your Understanding
A developer tries to delete a user by setting the hash bit indices back to 0. Why is this forbidden in a standard Bloom Filter?
You need to calculate the exact unique number of monthly active users (500 Million users) across 10,000 live channels with minimal RAM. Which data structure should you choose?