The one-line version
Many computers pretend to be one. They can't share a clock, and none of them can know for certain, right now, whether another has died or is merely slow. Everything on this page follows from those two facts.
The question to ask of any component is: what happens when it is slow, down, partitioned, duplicated or reordered, and who decides? If a design document can't answer that for each dependency, it hasn't finished.
The vocabulary
These are the terms the rest of the atlas uses without stopping to define them.
| Term | What it means |
|---|---|
| Timeout | How long a caller waits before treating a call as failed. It cannot tell "dead" from "slow" |
| Retry | Sending the call again after a failure. Turns one request into several |
| Idempotency keyA value attached to a request so the receiver can spot a repeat and do the work only once. It makes retries safe.Distributed systems and time | A value the caller attaches so the receiver can recognise a repeat and do the work once |
| Lease | Permission held for a bounded time, such as "you are the leader for the next ten seconds" |
| Quorum | A majority of members. Two majorities of the same group always overlap |
| Term or epoch | A number that goes up every time leadership changes |
| Fencing tokenA number that rises each time leadership changes, sent with every write so storage can refuse writes from a leader that has been replaced.Distributed systems and time | The epoch, attached to each write, so storage can reject writes from a leader that has been replaced |
| Logical clock | A counter that orders events without trusting wall-clock time |
Mechanisms worth understanding properly
A timeout cannot distinguish dead from slow. This is worth memorising, because so much follows from it. The caller that times out doesn't know whether the work happened. If it retries, the work may happen twice.
Retries need idempotency. A retry turns one request into many, and only idempotency makes the repeats harmless. "Exactly-once delivery" is a phrase on broker feature pages, but what you can actually build end to end is at-least-once delivery plus idempotent or transactional processing. The effect happens once; the message may arrive several times.
Leader election grants a lease, and leases need fencing. A leader that pauses (a long garbage-collection stop, a VM migration, a laptop lid closing) can wake up still believing it holds the lease, and write. Storage has to reject writes that carry an old epoch. Without that, you get two leaders writing at once, which is the split brain that quorum alone does not prevent.
Quorum prevents two leaders because two majorities must overlap. In a five-member cluster, any two groups of three share at least one member, and that member won't vote for two leaders in the same term. This is why etcd, ZooKeeper and every Raft-based store want an odd number of members and stop accepting writes when a majority is unreachable.
Clock skew breaks more than log ordering. Leases, token exp and nbf checks, certificate validity and last-writer-wins conflict resolution all read a clock. NTP keeps clocks within milliseconds of each other on a good network; PTP gets below a microsecond where the hardware supports it. A host five minutes out rejects valid tokens as "not yet valid".
Patterns
These are the standard answers, and they work best together.
- Propagate deadlines. The caller's remaining time budget passes down the call chain, and each hop subtracts what it used. A service three hops down shouldn't spend thirty seconds on a request whose user gave up after two.
- Circuit breakers stop calls to a dependency that is failing, so it gets room to recover and the caller fails fast instead of piling up threads.
- Bulkheads cap the share of a pool any one tenant or dependency can use, so one bad neighbour can't take everything.
- Admission control rejects work early rather than queueing it forever. A fast "no" is kinder than a slow timeout.
- Retry at one layer only, with exponential backoff and jitter. The jitter matters as much as the backoff. Without it, every client that failed at the same moment retries at the same moment.
How it fails
Retry amplification. Each retrying layer multiplies the calls by one more than its retry count. Three layers each retrying three times is 4 × 4 × 4 = 64 times the normal load, arriving exactly when that dependency is already struggling. The numbers page has the formula with the others.
Thundering herds at recovery. When Google's Service Control restarted in us-central1 in June 2025, the restarting tasks overloaded the Spanner database underneath them because they didn't back off with randomised delays, and full recovery in that region took up to about two hours forty minutes. Recovery is a load event. Plan for it.
Split brain without fencing. Covered above. The symptom is usually data that is quietly wrong, not an error.
Connection-pool exhaustion when a dependency slows down. A dependency that answers in five seconds instead of fifty milliseconds holds a hundred times more connections open. Slow is worse than down, because down fails fast and slow fills every queue on the way.
The trade-offs that don't go away
Consistency against latency. The PACELC framing is the useful one: if there is a partition, you choose between availability and consistency; else, even with no partition at all, you still choose between latency and consistency. A globally consistent write has to wait for a round trip to the other replicas. A fast local write has to accept that a reader somewhere else may not see it yet.
Synchronous against asynchronous replication. Synchronous replication gives you no data loss on failover (a recovery point objective, or RPO, of zero) at the price of slower writes. Asynchronous replication gives you fast writes, and an RPO equal to whatever replication lag you had at the moment of failure.
How the providers implement it
| Service | What it gives you |
|---|---|
| Google Spanner | External consistency across regions, using TrueTime's bounded clock uncertainty |
| Amazon DynamoDB | Eventually consistent reads by default, strongly consistent reads on request, and global tables that accept writes in several regions with asynchronous replication |
| Azure Cosmos DB | Five consistency levels to choose from, from strong to eventual |
| etcd | Raft consensus. The store behind every Kubernetes control plane |
Look the specifics up when you need them. They change, and the details (which operations are strongly consistent, in which modes) matter more than the headline.
The questions a senior architect asks
Where is the deadline set? Which layer owns retries, and which layers are forbidden from retrying? What is the idempotency key for each operation that changes something? What fences a stale leader? And what happens at ten times the normal latency, not just at zero availability?
The scenarios that exercise this page most are multi-region applications, event-driven integration and the zero-trust application.