DISTRIBUTED MINDS · Database Scaling
Interactive walkthrough

One database
can't take all the traffic.

Every technique on this page answers one question: when a single database server runs out of room, CPU, or write throughput — where does the next request go? Drag, click, and break things below to feel each answer instead of just reading it.

6scaling techniques
3live simulations
1viral-spike scenario
watching the ring…
click the ring to drop a key yourself · a live consistent-hashing ring — the technique most real distributed databases use to place data · full interactive version below
Ch.00

What actually happens when you "save to a database"

A database is a program with one job: store data safely, and hand it back fast. Every request — an app saving an order, a page loading a profile — follows roughly the same path.

1. Clientsends INSERT / SELECT
2. Query engineparses & plans it
3. Storage enginewrites to memory
4. Disk + indexpersists, updates lookups
5. Responseback to the client

One server can do this fast — for a while. It has a fixed amount of CPU, memory, and disk I/O, so it can only process so many requests per second before work starts piling up. Hit Start, then drag the slider and watch it happen live.

Single database server

sustainable capacity: 120 writes/sec
10%CPU
14mswrite latency (p99)
0%error rate
0queued writes
128,400 rows stored
Incoming traffic 40 req/s
fast (<250ms) slow (queued) dropped / timeout
⚠ Over capacity — writes are queuing and timing out. This is the exact problem the rest of this page solves.
Ch.01

Vertical vs. horizontal — the first fork in the road

Before picking an algorithm, every team hits this decision. It's cheap to delay, expensive to reverse.

Vertical scaling simple, finite

Give the same single database server more CPU, RAM, and faster disks. No code changes, no new failure modes — but every cloud provider has a biggest instance size, and you eventually hit it. It also means one machine is a single point of failure.

Horizontal scaling complex, unbounded

Spread data and load across many machines: more replicas for reads, more shards for writes and storage. Scales close to linearly, but now you're managing a distributed system — replication lag, partial failures, and routing logic all become your problem.

Ch.02

Replication: scale reads, survive failure

One leader takes writes; one or more followers copy its data and serve reads. Click write and watch how long it actually takes the copies to catch up.

Replication lag 800ms
Leader (accepts writes)
balance = $500
async replication →
Follower A (serves reads)
balance = $500
Follower B (serves reads)
balance = $500

Try this: click "Write", then immediately click "Read from a follower" before the bars finish filling. That gap is why "read-your-own-writes" bugs exist — the classic symptom of leader–follower replication.

Single-leader (leader–follower) common default

All writes go through one leader; followers replicate asynchronously (fast, can lose the last few writes on crash) or synchronously (safe, but writes wait on the slowest follower). Used by Postgres, MySQL, MongoDB replica sets.

Multi-leader write anywhere

Multiple leaders (often one per data center) each accept writes and replicate to each other. Great for multi-region latency, but two leaders can accept conflicting writes to the same row at once — you need a conflict-resolution strategy (last-write-wins, CRDTs, app-level merge).

What this actually looks like on real servers

The "balance = $500" box above is a simplification. Nothing that clean gets sent over the wire. What a leader really ships to its followers is its write-ahead log (WAL) — an append-only file of raw change records — which each follower replays against its own copy of the data files. Click the button and watch a real write travel as log entries, not values.

db-primary-01
10.0.1.10:5432 · role: primary
postgres (pid 4021) — accepting reads + writes
disk — /var/lib/postgresql/data
base/16384/ — table & index pages
pg_wal/ — write-ahead log (append-only)
idle — no writes yet
WAL stream
db-replica-01
10.0.1.11:5432 · role: standby (read-only)
postgres (pid 4033) — WAL receiver + replaying
pg_wal/ — replayed from stream
idle — no writes yet
db-replica-02
10.0.1.12:5432 · role: standby (read-only)
postgres (pid 4041) — WAL receiver + replaying
pg_wal/ — replayed from stream
idle — no writes yet

Each entry is one WAL record — the actual unit shipped over TCP. A replica doesn't receive "balance = $500"; it receives something like "at byte offset X, change these bytes on this page" and replays that against its own on-disk copy. That's why a replica can briefly be a few records behind: it's a receive-and-replay pipeline, not a shared value.

Want to actually set this up? step-by-step guide

A separate walkthrough of the real commands — creating the replication user, editing pg_hba.conf and postgresql.conf, taking a base backup, and verifying it's streaming — plus a Docker Compose version to try locally.

Same idea, different names

"Leader/follower," "primary/replica," and "master/replica" all describe the same role split — which node accepts writes vs. which one copies it. The exact term just depends on which database you're reading about.

DatabaseWrite node is calledCopy node is called
PostgreSQLprimarystandby / replica
MySQLsource (formerly "master")replica
MongoDBprimarysecondary
Redismasterreplica
Kafkaleader (per partition)follower
Ch.03

Partitioning: when one machine can't hold the data

Replication copies the whole dataset everywhere. Partitioning (sharding) splits the dataset itself across machines, so each one only holds a slice. The question every strategy answers differently: given a key, which shard owns it?

Range-based

Sort keys, split into contiguous ranges (IDs 1–1000 → Shard 0, 1001–2000 → Shard 1…). Great for range queries ("give me orders from last week"), but sequential keys concentrate all current writes on one shard — see the simulation below.

Hash-based

shard = hash(key) % N. Spreads writes evenly — no hot shard. The cost: adding or removing a shard changes the modulus, so almost every key remaps to a new shard at once. Massive, unnecessary data movement.

Consistent hashing

Place shards and keys on the same hash ring; a key belongs to the next shard clockwise. Adding or removing a node only remaps the keys between it and its neighbor — everyone else is untouched. Used by Cassandra, DynamoDB, Riak, CDN request routing.

Directory-based (lookup service)

A separate service keeps an explicit key → shard map. Maximum flexibility (move any single key any time, rebalance by hand) at the cost of an extra hop and a new single point of failure — the directory itself usually needs to be replicated.

Simulation — sequential IDs, range vs. hash

Watch where each new row lands as IDs increment one by one (like an auto-increment primary key, the most common real-world case).

Range sharding — 4 shards
Hottest ÷ average load:
Hash sharding — 4 shards
Hottest ÷ average load:

Interactive — consistent hashing ring

Add or remove nodes and watch how few keys actually move. Click any node to remove it. Type a key below to see exactly where it routes.

0%keys remapped just now (consistent hashing)
~87%keys that would remap with plain hash % N

Federation split by function

Instead of splitting one table across shards, split by function: a Users database, an Orders database, a Payments database — each on its own server. Simple to reason about, but you lose cross-domain joins and transactions; the application has to stitch data back together.

Denormalization + caching trade writes for reads

Duplicate data across tables/shards so reads don't need joins across machines, and put a cache in front of hot reads entirely. Writes get more expensive and consistency gets harder — but most apps are read-heavy, so the trade usually wins.

Ch.04

Real application behavior — a checkout service under load

Users are sharded by user_id across 3 primaries, each with one async replica. Watch what happens when traffic is normal — and when one user suddenly goes viral.

Clientcheckout
Load Balancerround-robin
App Serverstateless
Shard Routerhash(user_id)
Primary DBwrites
Shard A · primary 0 req/s
0
↳ replica lag: 0ms
Shard B · primary 0 req/s
0
↳ replica lag: 0ms
Shard C · primary 0 req/s
0
↳ replica lag: 0ms
42msp50 latency
118msp99 latency
0%error rate

The lesson: sharding by user_id spreads out normal traffic beautifully, but it can't save you from one abnormally hot key — every request for that one user still lands on the same shard. Fixing it needs a different move: cache their reads, add a replica to absorb read load, or split that one user's data further.

Ch.05

The CAP theorem — pick your trade-off

During a network partition, a distributed database can guarantee at most two of Consistency, Availability, and Partition tolerance. Click two corners.

Pick two

Click any two corners of the triangle

Partition tolerance means the system keeps working even when network links between nodes drop or lag badly. In any real distributed database — deployed across more than one machine — partitions will happen sooner or later. So P usually isn't optional; the real everyday choice is between C and A.

Ch.06

Decision cheat sheet

Print this in your head before your next design interview.

TechniqueSolvesTrade-offReach for it when
Vertical scalingNot enough CPU/RAM/diskHard ceiling, single point of failureEarly stage, simplest possible fix
Read replicasToo many reads for one serverReplication lag, stale readsRead-heavy workloads (most CRUD apps)
Leader–followerReads + durabilitySingle write bottleneck remainsDefault choice for most systems
Multi-leaderMulti-region write latencyConflict resolution complexityUsers write from multiple regions
Range shardingStorage + range-query performanceHot shard on sequential keysTime-series / range-scan heavy data
Hash shardingEven write distributionMassive remap on resizeFixed, rarely-resized cluster
Consistent hashingEven distribution + cheap resizeExtra routing layer to build/runClusters that grow/shrink often
Directory-basedMaximum placement flexibilityDirectory is a new dependencyNeed per-key control (tenant isolation)
FederationOne giant DB doing everythingLoses cross-domain joins/transactionsClear functional boundaries exist
Denormalize + cacheExpensive joins on hot readsWrite cost, cache invalidationRead-to-write ratio is high