Unique ID Generator (Twitter Snowflake)
How to mint 4 million sortable IDs per second per node with zero coordination. Master the 64-bit Snowflake layout, clock-drift guards, sequence exhaustion, node ID assignment, and UUIDv7.
π« 1. The Real-World Intuition: 50 Hospitals, No Phone Lines
You run a national health system with 50 hospitals. Every newborn needs a certificate number that is unique across the whole country β and the hospitals cannot call each other. There is no central registry to ask. Babies arrive at 3 a.m. during a network outage and still need a number, instantly.
Each hospital keeps its own notepad: 1, 2, 3, 4β¦ It works perfectly in isolation. Then you merge the national database and discover that certificate #1001 belongs to a baby in Mumbai and a baby in Delhi.
Result: Collisions everywhere. Your "unique" primary key is a duplicate 50 times over, and you cannot tell which record is real.Give every hospital a permanent 3-digit code, then stamp each certificate as20260824-042-0007β the date, the hospital, and a small counter for babies born the same day.
Notice why the second scheme needs no phone call. Two hospitals can never collide because their hospital code differs. One hospital can never collide with itself because either the date differs, or β if two babies arrive the same day β the counter differs. Every possible collision is closed off by construction.
π₯ 2. Why the Obvious Answers Break
Before reaching for Snowflake, understand the four schemes it replaces β and the specific way each one dies.
AUTO_INCREMENTThe database hands out 1, 2, 3β¦ under a lock. Flawless on one node, and genuinely the right answer until you shard.The failure: the counter lives inside one database. That database is now a single point of failure for every write in your system, and every ID request pays a network round trip. Shard it and both shards start at 1.
Set auto_increment_increment = 2 and give server A offset 1 (odd IDs) and server B offset 2 (even IDs). Clever, and it works.The failure: the step size is baked into the topology. Adding a third server means re-planning every offset across the fleet, and IDs from different servers interleave unpredictably so they are no longer time-ordered.
Two dedicated MySQL boxes do nothing but REPLACE INTO Tickets64 then return LAST_INSERT_ID(), one serving odd numbers and one even.The failure: it is still a network hop on the critical path of every insert. You have traded a database bottleneck for a smaller, faster bottleneck β and you still cannot generate an ID while offline or partitioned.
122 random bits, generated locally, no coordination. Collision risk is negligible β you would need to mint billions of UUIDs before a collision became as likely as a meteor strike.The failure is not collisions β it is index locality. Random IDs land at random positions in a B+Tree. Instead of appending to the rightmost leaf page (which stays hot in the buffer pool), every insert dirties a different page: page splits, fragmentation, and a buffer pool that thrashes because the working set is now the entire index instead of its right edge.
| Scheme | Size | Time-Sortable? | Coordination Needed? | Index Locality |
|---|---|---|---|---|
| AUTO_INCREMENT | 8 bytes | β Perfectly | β Yes β every insert | β Excellent |
| Ticket Server | 8 bytes | β Perfectly | β Yes β network hop | β Excellent |
| UUID v4 | 16 bytes (36 as text) | β Never | β None | β Terrible |
| Snowflake | 8 bytes | β Roughly | β οΈ Node ID only, once | β Excellent |
| UUID v7 | 16 bytes | β Roughly | β None | β Very good |
𧬠3. Anatomy of a 64-Bit Snowflake ID
Snowflake takes the hospital scheme and compresses it into a single 64-bit signed integer β small enough to be a primary key, an int64, and a Kafka partition key all at once.
241 = 2,199,023,255,552 ms.
Γ· 86,400,000 ms/day = 25,451.7 days
Γ· 365.25 = 69.7 years of headroom.
210 = 1,024 distinct node IDs (0β1023). Twitter originally split this as 5 bits datacenter + 5 bits worker.
212 = 4,096 IDs per millisecond per node
= 4,096,000 IDs/second from a single node.
The custom epoch is what makes 41 bits enough. Counting milliseconds from 1970 would already have burned ~56 years of the budget. Twitter instead counted from 1288834974657 ms β 2010-11-04T01:42:54.657Z β buying the full 69.7 years from that date, which exhausts in July 2080.
// PACK β an ID minted on node 42, 8th ID of that millisecond wall clock = 1787572800000 // 2026-08-24T12:00:00.000Z EPOCH = 1288834974657 // 2010-11-04T01:42:54.657Z delta = 1787572800000 - 1288834974657 = 498737825343 ms id = (delta << 22) | (node << 12) | sequence = (498737825343 << 22) | (42 << 12) | 7 = 2091858055787618311 // UNPACK β pure shifts and masks, no database lookup needed timestamp = (id >> 22) + EPOCH = 1787572800000 // 2026-08-24T12:00:00.000Z node = (id >> 12) & 1023 = 42 sequence = id & 4095 = 7
The shift amounts fall straight out of the layout: the sequence occupies the low 12 bits, so the node shifts left by 12; the node occupies 10 bits above that, so the timestamp shifts left by 10 + 12 = 22. The masks are just (1 << 12) - 1 = 4095 and (1 << 10) - 1 = 1023.
long, a Go int64, and a Postgres BIGINT are all signed β setting the high bit produces a negative ID. Leaving it zero guarantees every Snowflake ID is a positive integer that sorts correctly in every database on earth.A Snowflake ID is 19 digits. JavaScript numbers are IEEE-754 doubles, exact only up toNumber.MAX_SAFE_INTEGER = 9007199254740991 β 16 digits. A Snowflake ID is roughly 232Γ larger than the largest integer JavaScript can represent exactly. So JSON.parse does not error β it rounds, and quietly hands your front-end the wrong ID.
// The server sends this JSON: { "id": 2091858055787618311 } // The browser parses it: JSON.parse(body).id // => 2091858055787618300 // β last 2 digits destroyed! // The fix: serialize IDs as STRINGS on the wire. { "id": "2091858055787618311" } // β survives intact
Symptoms in the wild: two different records appear to share an ID, clicking a row opens the wrong record, and a permalink 404s β while the API returns perfectly correct data. This is why the Twitter API has always returned both id and id_str.
Snowflake IDs are k-sorted, not globally ordered. Within one node, IDs strictly increase. Across nodes, two IDs minted in the same millisecond order by node ID, not by the true instant β and clocks drift by a few milliseconds anyway. So Snowflake gives you excellent index locality and a reliable coarse chronological sort, but never treat "id_a < id_b" as proof that A happened before B. If you need true ordering, you need logical clocks or consensus, not timestamps.
β° 4. The Three Ways Snowflake Fails in Production
The bit layout is the easy part β you can draw it in an interview in thirty seconds. What separates someone who has operated an ID service from someone who has only read about one is knowing these three failure modes.
Snowflake's uniqueness rests entirely on the assumption that the wall clock never moves backward. But NTP does exactly that: when it finds the local clock ahead of the true time, it can step the clock back. The generator then re-enters a millisecond it has already used, restarts the sequence at 0, and re-emits IDs it already handed out. You get duplicate primary keys β the one failure a unique ID generator must never have.
Cloud VMs are especially exposed: live migration, VM pause/resume, and snapshot restore can all warp the guest clock in a single jump.
| Mitigation | How It Works | The Trade-Off |
|---|---|---|
| Refuse & throw | Track lastTimestamp; if now < last, raise an exception. Twitter's original behaviour. | Chooses correctness over availability β the service returns errors until the clock recovers. |
| Spin until caught up | Block in a loop until the wall clock passes lastTimestamp. | Fine for a 5 ms jump; a 10-minute jump hangs every caller. Always cap the wait. |
| Monotonic clock source | Take the wall clock once at startup, then advance using a monotonic timer that NTP cannot rewind. | Immune to steps, but slowly diverges from real time β decoded timestamps become approximate. |
| Coarser resolution | Sonyflake-style: use 10 ms ticks instead of 1 ms, spending the saved bits on lifetime and nodes. | Absorbs small drift and extends lifespan, at the cost of per-node throughput. |
| Clock-free mode | Alibaba Seata-style: treat the ID as a pure monotonic counter, only consulting the clock at startup. | Cannot drift at all, but IDs no longer decode to a trustworthy wall-clock time. |
Operational note: configure time sync to slew (adjust gradually) rather than step (jump) β chronyd with makestep restricted, or ntpd -x. Never run a cron job that calls ntpdate on a host that mints IDs.
12 bits give you 4,096 IDs per millisecond. Ask for the 4,097th within the same millisecond and the sequence wraps back to 0 β which would duplicate the IDs you just issued. The correct behaviour is to busy-wait until the clock ticks over to the next millisecond, then reset the sequence:
if (ts == lastTimestamp) { sequence = (sequence + 1) & 4095; // SEQ_MASK if (sequence == 0) // all 4096 slots used this ms ts = tilNextMillis(lastTimestamp); // β spin here } long tilNextMillis(long last) { long ts = System.currentTimeMillis(); while (ts <= last) ts = System.currentTimeMillis(); return ts; }
This is correct but it creates a latency cliff: p50 stays at nanoseconds while p99.9 jumps to a full millisecond of spinning. At 4M IDs/sec you are riding that cliff constantly. The fix is horizontal β add nodes, since each one gets its own 4,096/ms budget.
Everything above assumes each generator has a unique node ID. If two pods both believe they are node 3, they emit identical IDs β silently, forever, with no error anywhere. This is the problem interviewers actually probe, because it is the one with no clean answer.
| Approach | Pros | Cons | Verdict |
|---|---|---|---|
| Static config file | Zero dependencies, trivial to reason about. | One copy-pasted config and you have silent duplicates. Cannot autoscale. | β Only for fixed, hand-managed fleets. |
| ZooKeeper ephemeral sequential znode | Strong guarantee; the node ID is released automatically when the process dies. | Operating a ZooKeeper ensemble; ID churn on every restart. | β The classic, battle-tested answer. |
| etcd / Consul lease | Same guarantee with a modern API and TTL-based expiry. | Still an external dependency on the startup path. | β Preferred if you already run etcd. |
| Kubernetes StatefulSet ordinal | Free β the pod is already named gen-0, gen-1; no extra system to run. | Ordinals are reused on scale-down then scale-up; capped at 1,024 replicas. | β οΈ Elegant, but read the trap below. |
| Derive from private IP / MAC | No coordination at all. | 10 bits cannot hold an IP β you truncate, and truncation collides. | β A collision waiting to happen. |
The StatefulSet trap: scale 8 β 3 β 8 and the new gen-5 inherits the node ID the old gen-5 was using seconds ago. If the old pod's clock was even slightly ahead, the new pod re-issues IDs from a millisecond the old pod already consumed. A node ID must not be reusable until longer than your maximum clock-skew window β which is exactly what a lease TTL gives you and a bare ordinal does not.
tilNextMillis spins. Rising = you need more nodes.π 5. The Modern Alternatives: UUIDv7, ULID, KSUID
Here is the honest 2026 answer that many system design write-ups skip: most teams should not build a Snowflake service. UUIDv7 β standardised in RFC 9562, which obsoleted the old RFC 4122 in May 2024 β gives you time-ordered, index-friendly IDs with no node IDs, no ZooKeeper, and no clock-drift runbook. Reach for Snowflake when you specifically need 64 bits (half the storage, cheap in every index and foreign key) or more than 4,096 IDs per millisecond per process.
128 bits: a 48-bit big-endian Unix millisecond timestamp, then the 4-bit version (0111), 12 bits of randomness, the 2-bit variant (10), and 62 more random bits. Time in the high bits means it sorts and indexes almost as well as Snowflake, while the 74 random bits remove the need for any node coordination.
48 bits of milliseconds from 1970 lasts until the year 10899 β no epoch cliff. Native uuidv7() landed in PostgreSQL 18, and every major language has a library. UUIDv6 is a legacy-compatible reordering of v1; UUIDv8 is a free-form slot for custom layouts.
128 bits as 26 characters of Crockford Base32: 48-bit millisecond timestamp + 80 random bits. Its trick is that the text form sorts lexicographically β perfect for object-store keys and log filenames. A monotonic variant increments the random part within the same millisecond.
20 bytes as 27 Base62 characters: a 32-bit second-resolution timestamp (custom 2014 epoch) + 128 random bits. Segment's design. Second resolution means it will not order events inside the same second β fine for records, wrong for a high-rate feed cursor.
12 bytes: 4-byte second timestamp + 5 random bytes per process + a 3-byte counter. The 5 random bytes replaced the old machine-ID + PID scheme, which leaked host details and collided in containers. Same three ingredients as Snowflake, in 96 bits.
64 bits: 41-bit ms timestamp + 13-bit logical shard ID + 10-bit sequence, minted by a PL/pgSQL function inside Postgres. Embedding the shard ID means any ID tells you which shard to query β no lookup table. Proof you may not need a separate ID service at all.
| You need⦠| Choose | Because |
|---|---|---|
| A sharded OLTP primary key, minimum storage | Snowflake | 8 bytes instead of 16 β it multiplies across every secondary index and foreign key. |
| A primary key with the least operational burden | UUID v7 | Time-sortable and coordination-free. No node registry, no drift runbook, no epoch cliff. |
| An ID generated offline on a client or mobile device | UUID v7 / ULID | Enough entropy to be safe without ever contacting a server to claim a node ID. |
| A sortable cursor for an infinite feed | Snowflake / ULID | Millisecond resolution orders events inside a second; KSUID's 1-second granularity cannot. |
| A key in an object store or a filename | ULID | The 26-char text form sorts lexicographically, so listings come back in time order. |
| A public-facing ID in a URL | Opaque random ID | Never expose a sortable internal key β see the security warning below. |
A Snowflake ID is reversible. Anyone holding one can extract the exact millisecond the record was created β no key required, just a right-shift. Worse, IDs from two points in time reveal your growth rate: sample /orders/<id> on Monday and Friday, subtract, and a competitor knows how many orders you processed that week. This is the German tank problem, and sequential IDs make it trivial.
Sequential IDs also invite enumeration attacks: /invoice/1002 implies /invoice/1001 exists, turning one weak authorization check into a full data breach.The standard fix: keep the sortable Snowflake as your internal primary key, and expose a separate opaque public ID β a random UUIDv4 or a signed slug β in URLs and API responses. Internal locality and external opacity are different jobs; use two columns.
π» 6. Production Snowflake Generator (Multi-Language)
A complete, thread-safe generator in 5 languages. Every version implements the same four essentials: named bit constants, a lock, the sequence-rollover busy-wait, and the clock-drift refusal.
# Python 3.10+ β thread-safe Twitter Snowflake generator import time, threading class SnowflakeGenerator: EPOCH_MS = 1288834974657 # 2010-11-04T01:42:54.657Z (Twitter's epoch) SEQUENCE_BITS = 12 NODE_BITS = 10 NODE_SHIFT = SEQUENCE_BITS # 12 TS_SHIFT = NODE_BITS + SEQUENCE_BITS # 22 SEQ_MASK = (1 << SEQUENCE_BITS) - 1 # 4095 MAX_NODE = (1 << NODE_BITS) - 1 # 1023 def __init__(self, node_id: int): if not 0 <= node_id <= self.MAX_NODE: raise ValueError(f"node_id must be 0..{self.MAX_NODE}") self.node_id = node_id self._seq = 0 self._last_ts = -1 self._lock = threading.Lock() # the GIL alone is NOT enough def _now(self) -> int: return int(time.time() * 1000) def next_id(self) -> int: with self._lock: ts = self._now() if ts < self._last_ts: # CLOCK DRIFT GUARD raise RuntimeError(f"clock moved back {self._last_ts - ts}ms") if ts == self._last_ts: self._seq = (self._seq + 1) & self.SEQ_MASK if self._seq == 0: # all 4096 slots used this ms ts = self._til_next_ms(self._last_ts) else: self._seq = 0 self._last_ts = ts return ((ts - self.EPOCH_MS) << self.TS_SHIFT) \ | (self.node_id << self.NODE_SHIFT) \ | self._seq def _til_next_ms(self, last: int) -> int: ts = self._now() while ts <= last: ts = self._now() return ts
TS_SHIFT is 22, not 41 β a field shifts left by the number of bits below it (10 + 12), never by its own width.BigInt internally and return .toString() at every boundary.π§ͺ 7. Interactive Lab: Live 64-Bit Snowflake Visualizer
Generate, Decode & Break a Snowflake ID
π‘ What to notice: consecutive IDs differ only in their last few digits β the timestamp and node bits are identical, so every insert lands right next to the previous one in the B+Tree. That is the index locality Snowflake buys you.
π― 8. Test Your Understanding
At 02:00 your ID service starts throwing duplicate-primary-key errors for 700 milliseconds, then recovers on its own. The only correlated event in the logs is an NTP synchronization. What happened, and what is the fix?
Your React app lists orders. Two clearly different orders display the same ID, and clicking one opens the wrong record. You curl the same endpoint and the JSON is completely correct. Where is the bug?
You derive each generator's node ID from its StatefulSet ordinal (gen-0 β node 0, and so on). An autoscaler scales 8 β 3 β 8 pods within a minute, and duplicate IDs appear. Why?