The Numbers Architects Reason With

Why do the arithmetic

Most architecture arguments are settled faster with a number than with an opinion. "Will multi-region help?" turns into "what is our RPO, and what is replication lag at peak?" "Can we self-host the model?" turns into "what is our utilisation going to be?" This page collects the thirteen calculations the rest of the atlas leans on. None of them needs more than a calculator, and every one of them has changed a design decision somewhere.

Where a figure is illustrative rather than measured, it says so. The point is the shape of the relationship, not the vendor's benchmark.

Light in fibre

Light travels through fibre at about 2 × 10⁸ metres a second, two thirds of its speed in a vacuum. That's about 0.5 ms to cover 100 km one way, so roughly 1 ms of round trip for every 100 km of fibre path. Treat it as a floor: real routes are longer than the great-circle distance, and every router and amplifier on the way adds a little.

What it means in practice: a protocol that makes ten round trips between continents, at about 80 ms each, has spent 0.8 seconds before it has done any work. Chatty protocols decide where things can be placed. A database that needs three round trips per transaction cannot sit an ocean away from its application.

Availability in series and in parallel

In series, every dependency has to be up, so you multiply their availabilities together. Five dependencies at 99.9% each give 0.999⁵ = 0.99501, or 99.50%. That's about 43.8 hours of downtime a year, against 8.8 hours for any one of them alone.

In parallel, with redundant copies that fail independently, the system is down only when all of them are, so you multiply their unavailabilities together and subtract the result from 1. Two replicas at 99.9% give 99.9999%, but only if their failures are independent and failover is perfect.

That condition is where the arithmetic usually lies. Correlated failure (a shared control plane, a shared configuration push, a shared identity provider) collapses the parallel formula back towards the series one. Two zones behind one regional control plane are not two independent 99.9% systems for a control-plane fault. The maths is only as good as the independence assumption, which is why the failure page spends so long on what components share.

Static-stability capacity

To survive losing one of n zones without launching anything new, each zone needs capacity for 1/(n−1) of peak. The total you provision is n/(n−1) times peak:

• 3 zones: 3/2 = 1.5 times peak
• 2 zones: 2/1 = 2 times peak
• 4 zones: 4/3 = about 1.33 times peak

That's the price of not depending on the control plane during a zonal failure, and it explains why three zones is the usual answer: the third zone costs less than the extra headroom two zones would need.

Queueing: why utilisation fights latency

For a single server with random arrivals (the M/M/1 model), the mean time a request spends in the system is S / (1 − ρ), where S is the service time and ρ is utilisation, the fraction of time the server is busy.

• at ρ = 0.5, time in system is 2S
• at ρ = 0.8, it's 5S
• at ρ = 0.9, it's 10S
• at ρ = 0.95, it's 20S

Real systems aren't exactly M/M/1, but the curve has the same shape almost everywhere: gentle until about 70%, then a wall. This is the tension at the centre of GPU economics. Idle GPUs are expensive, so the finance case pushes utilisation up. Latency targets push it down. Batching changes S, which moves the curve. Admission control is how you choose your point on it, by refusing work before the queue grows.

Little's law and fan-out

Little's law says that the average number of requests in a system, L, equals the arrival rate, λ, times the average time each spends there, W: L = λW. At 200 requests a second and 4 seconds of mean latency, 800 requests are in flight at any moment. That one number sizes your connection pools, your gateway's concurrency limit and, for a language model, the KV-cache memory you need.

Fan-out turns rare slowness into common slowness. If a request calls 100 services in parallel and each has a 1% chance of being slow, the chance that all of them are fast is 0.99 to the power of 100, about 37%. So the chance that at least one is slow is 63%. Most of your users now see the tail. This is the arithmetic behind the advice to keep microservice fan-out shallow.

Retry amplification

Each layer that retries multiplies the attempts by one more than its retry count. Three layers that each retry three times give 4 × 4 × 4 = 64 times the load on the bottom dependency during an outage.

The rule that follows: retry at one layer only, use a retry budget rather than a fixed count, and add jitter so the retries don't arrive together.

The revocation window

When you remove someone's access, how long can they still use it? Call that W_revoke. It's the sum of every delay between the decision and its enforcement:

Add together the remaining token validity (or the event-revocation delay, at resources that support it), the TTL of any decision cache, the group-sync lag and the index ACL-sync lag.

Worked example: a resource that doesn't take part in continuous access evaluation accepts tokens with 60 to 90 minutes of life, the policy decision point caches for 5 minutes, and group membership syncs every 30 minutes. The worst case is about two hours.

This is the number to set per data class. "Bounded staleness" in a design document means nothing until it is written as a figure like this one. The identity page draws it.

Error budgets

An error budget is (1 − SLO) × the measurement window. Over 30 days:

At 99.99% there isn't time for a person to be paged, open a laptop and diagnose anything. Recovery has to be automatic, or the system has to fail static. Choosing four nines is choosing an architecture, not a target.

KV-cache sizing

A transformer stores keys and values for every token it has seen, for every layer. The bytes needed per token are:

2 (one for K, one for V), multiplied by the number of layers, the number of KV heads, the head dimension and the bytes per element

For an illustrative 70B-class model using grouped-query attention, with 80 layers, 8 KV heads, a head dimension of 128 and FP16 values (2 bytes): 2 × 80 × 8 × 128 × 2 = 327,680 bytes, or 320 KiB per token. An 8,000-token sequence needs about 2.6 GB of KV cache.

The weights take about 70 billion × 2 bytes = 140 GB at FP16, or about 70 GB at FP8. On two 80 GB GPUs:

• at FP16 the weights leave roughly 20 GB, less activations and overhead, so about six or seven concurrent 8k sequences
• at FP8 the weights leave roughly 90 GB, so thirty or more

Quantisation and grouped-query attention buy batch size, and batch size is throughput. Treat the parameters here as assumptions and look them up for the model you actually run.

Cost per token, self-hosted

Divide the GPU's hourly price by the tokens it produces in an hour, scale to a million tokens, then divide by utilisation.

Using the AWS P5 Capacity Block rate effective 7 October 2026, $5.970 per accelerator-hour, and an assumed 1,500 output tokens a second per GPU:

• at 100% utilisation: 5.970 ÷ (1,500 × 3,600) × 10⁶ ≈ $1.11 per million tokens
• at 30% utilisation: about $3.68 per million tokens

Idle time dominates. Compare self-hosting against API prices at the utilisation you will actually achieve, not the one on the slide.

Cost per request, through an API

The cost of a request is its input tokens at the input price, plus its output tokens at the output price.

Retrieval-augmented requests are dominated by input: a typical shape is 6,000 tokens of retrieved context and 500 tokens of answer. Halving the retrieved context often saves more than switching to a cheaper model. Cached prompt prefixes change the input price, so check what your provider charges for them.

RTO and RPO

• With asynchronous replication, the recovery point objective is at least the replication lag at the moment of failure.
• With backups, the RPO is at most the backup interval: 24 hours for a nightly backup.
• The recovery time objective is the sum of detection, decision, DNS TTL and propagation, warm-up, and restoring dependencies in the right order.

DNS failover, worked through: three failed health checks at 30-second intervals, and then a 300-second TTL, come to about 6.5 minutes, before you count the clients that ignore TTLs.

Cache hit rate

Backend load is (1 − h) × λ, where h is the hit rate. Going from a 95% hit rate to 90% doubles the load on the backend, from 5% of requests to 10%. And when many entries expire at once, the misses arrive together as a herd, so give TTLs some jitter.