50 Servers, Zero Collisions

The problem, stated plainly

A URL shortener has exactly one job that sounds trivial and isn't: turn a long link into a short code that nobody else has ever used, and never will.

Say the traffic level. 100 million links. 50 servers behind a load balancer, each one taking requests independently. Now ask the question that actually matters:

How does a server know a code is free without asking anyone?

The obvious answer — keep a central database of every code that's been issued, and check it before handing out a new one — works. It also quietly becomes the whole system's ceiling. Every one of those 50 servers now depends on one database being up, being fast, and being reachable, for every single request. Add more servers and you don't get more throughput, you get more contention on the one resource everyone has to share. The database that was supposed to guarantee correctness becomes the thing that decides how fast the whole product can grow.

So the real question isn't "how do we check for collisions fast enough." It's sharper than that:

What if there were nothing to check?

Why "just add a cache" doesn't solve it

The instinct is to keep the central database but hide its latency behind a cache, or shard it, or replicate it. All of these make the symptom smaller. None of them remove the actual constraint, which is this: as long as uniqueness is decided at request time by comparing against everyone else's history, some form of coordination is unavoidable. A cache still needs to be invalidated correctly across 50 servers. A shard still needs a router that knows which shard owns which key — which is itself a shared piece of coordination logic. You can make the bottleneck faster. You can't make it disappear, because the design still assumes servers need to ask.

The alternative is to stop asking. Decide, in advance, that it's structurally impossible for two servers to produce the same code — not unlikely, not checked, impossible — and then nobody needs to check anything, ever.

The naive shape of the problem

Every arrow in that picture is a dependency. If the database is slow, all 50 servers are slow, at the same moment, for the same reason. If it's briefly unreachable, every server is stuck, not just one. The system's reliability is capped at the reliability of its single busiest component — the one everyone was told to check with.

Reframing the question

Here's the move that unlocks the rest of this design: uniqueness doesn't have to be verified. It can be guaranteed by construction.

Instead of "check if this code is free," the question becomes: can we design the codes so that it's mathematically impossible for two servers to ever generate the same one — without either server knowing the other exists?

That reframing turns a runtime coordination problem into a one-time setup decision. And once it's a setup decision, it can be made once, at deploy time, and never revisited on the request path again.

The actual answer: give every server its own signature

Each of the 50 servers is assigned a fixed, permanent ID — an ordinary number from 0 to 49 — when it's deployed. Nobody negotiates this at runtime. It's handed out once, the same way you'd hand out employee badge numbers before anyone starts their shift.

From that point on, every code a server generates carries its ID embedded inside it, like a watermark. The rest of the code comes from a counter that server keeps entirely in its own memory — a number it increments every time it hands out a new code. That counter is never written to a shared database, never synced, never even visible to any other server.

Two servers can only ever produce the same final number if every part of it matches — and the server-ID portion is permanently different between them. Server 7 could never produce a number that Server 12 could also produce, no matter what either of their counters is doing, because the last few bits of every number Server 7 emits always say "7," and the last few bits of every number Server 12 emits always say "12." It's not that a collision is unlikely. It's that the two servers are drawing from disjoint sets of numbers, by construction. There is no set of coincidences that could make them overlap.

That single design decision is what removes the central database from the picture entirely. Nobody has to check anything, because there's nothing left to accidentally collide.

"But doesn't that make the codes predictable?"

Good instinct — and yes, on its own it would. If Server 7's counter goes 1, 2, 3, 4, then its raw codes would climb in an obvious, guessable order. Anyone who saw one short link could start incrementing it and walk through every other link the same server had ever issued.

That's what the scramble step in the diagram above is for. Before a raw number becomes a short code, it's run through a reversible mathematical shuffle — the kind of operation where every input maps to exactly one output and every output maps back to exactly one input. Because it's one-to-one in both directions, it can reorder the numbers however chaotically it likes without ever merging two different numbers into the same output. Unpredictable on the outside, still collision-free underneath, for free.

Following one request all the way through

Notice the asymmetry in that diagram, because it's the whole point: creating a code is a local decision Server 7 makes entirely on its own — no coordination, no lookup, no waiting on anyone. Resolving a code back to a URL is a different problem, because a redirect for a link Server 7 created might land on Server 23 instead, depending on which one the load balancer happens to route to that moment. Server 23 has no idea what Server 7's counter is doing, and it doesn't need to — but it does need somewhere shared to look up what 5p7OCYF points to.

The honest boundary of this design

This is worth being direct about, because it's the natural next question and skipping it would be dishonest: generating a unique code needs zero coordination. Looking one up later still needs a shared, queryable store — a real database, not a local map on one server. Those are two different problems wearing the same URL. The architecture here removes the database from the write path entirely, and keeps it only for reads, which is a very different load profile: one shared store getting simple key lookups, instead of every server racing to ask "is this taken?" before every single write.

There's a second honest gap: a server's counter lives in memory. If that server crashes and restarts, the counter resets to zero — and it could, in theory, reissue a code it already gave out before the crash. Solving that doesn't require going back to a central database; it just means the counter needs to survive a restart, typically by checkpointing it to local disk every so often instead of trusting memory alone.

Why this is worth the trade-off

The rhetorical question underneath all of this is really: what has to be true at request time, versus what only has to be true once, in advance?

Every collision-prevention scheme that checks at request time inherits the latency, the failure modes, and the scaling ceiling of whatever it's checking against. Every scheme that instead makes collisions structurally impossible — by partitioning the space up front and never overlapping it again — pays a one-time design cost and then gets to stop worrying about it. Fifty servers each holding a number nobody else can produce isn't a clever trick so much as a refusal to ask a question that never needed asking in the first place.

RELATED ARTICLES