A request that fans out to n independent backends and waits for all of them is fast only if every one of them is fast. If each call is inside its own p99 with probability 0.99, then all n are with probability 0.99ⁿ. That is 0.499837 at n = 69: a service with a fan-out of sixty-nine misses its per-call p99 on most requests, and nothing anywhere in it is broken.
Core ideas
The hard idea is a single exponent, and the reason it is hard is that the arithmetic is trivial and the conclusion is not one most people will believe before seeing it.
A request that issues n calls in parallel and needs all the answers finishes when the slowest returns. So “this request was fast” is the conjunction of n separate statements, and a conjunction of independent events multiplies: p × p × … × p = pⁿ.
0.99⁶⁹ = 0.499837, just under a half, and 0.99⁶⁸ = 0.504886, just over. So at a fan-out of sixty-nine, more than half of all requests contain at least one call outside its p99. The per-call p99 is now the whole request’s median, which is a statement about arithmetic rather than about anything being wrong.
A p90 per call breaks even at n = 7, a p95 at n = 14, a p99 at n = 69 and a p99.9 at n = 693. Each step of a factor of ten in the per-call tail buys roughly a factor of ten in the fan-out you can afford, which is the only lever there is apart from not fanning out.
The material
The exponent, the number sixty-nine, and the assumption the whole thing rests on.
A search page assembles itself from many services: one for results, one for advertisements, one for spelling, one for each of several personalisation signals, and a shard of the index for each partition of the corpus. They run in parallel, which is the right design — by the max rule of “Serial Sums, Parallel Maxes” the group costs what the slowest member costs rather than the sum. The question is what the slowest member costs.
Let a request issue n calls that are independent, and let each call complete within its own q-th percentile with probability p = q. Then the probability that every call is inside its percentile is pⁿ, and the probability that at least one is outside it is 1 − pⁿ.
The n at which pⁿ = ½ is the fan-out width at which the per-call q-th percentile has become the whole request’s median: half of all requests contain a call beyond it.
The proof is one line — independent events multiply — and the content is entirely in the numbers it produces.
Read the n = 100 row slowly. Every backend is meeting its latency objective, every dashboard is green, nobody is paged — and 63.40% of requests are waiting on a call that is outside the objective. There is no failure here to find, because there is no failure. The system is doing exactly what it was specified to do.
Those four rows are the whole strategy space. Reduce the fan-out, or push the per-call tail down by a factor of ten to buy a factor of ten in width, or stop requiring every answer — a request that can return with nine of ten results has changed the conjunction and is no longer in this arithmetic at all. There is also the option “Hedged Requests” takes, which is to change the per-call tail without changing the per-call service.
Push every call to a p99.9 — an expensive, real engineering achievement — and the break-even moves to n = 693. At a fan-out of 200 the probability that every call is inside its p99.9 is 0.999²⁰⁰, about 0.819, so 18% of requests still contain a slow call. That is a far better place to be than the 86.60% the same fan-out gives at a p99, and it is not zero and never will be.
Everything above assumes independence, and it is worth being precise about which direction reality departs from it. Real slowness has shared causes: a garbage-collection pause on a host that several calls land on, a saturated switch, a hot shard. Shared causes make slow calls arrive together, so fewer requests contain a slow call than 1 − pⁿ predicts — and the ones that do contain several. Independence is the assumption that spreads the damage as thinly and as widely as possible, and the real distribution is lumpier than it.
The misconception to name is the comfortable one: “each call is fast at its p99, so the request is.” The premise is about one call and the conclusion is about n of them, and the step between them is a multiplication that nobody performs because it does not look like there is anything to compute.
Interactive
pⁿ, and the width at which a per-call p99 becomes the request's median
Method
Four steps, and the third is the one that turns a design review around.
Only the ones it needs before it can answer. A call whose result is optional, or one that is fired and forgotten, is not in n. Shards count individually: a query against forty partitions is a fan-out of forty.
pⁿ is the probability that every call is inside its percentile. 1 − pⁿ is the probability that the request contains at least one that is not, which is the number worth quoting to anyone.
That is the width at which the per-call percentile has become the request’s median. Compare it with the fan-out you actually have; if yours is larger, the per-call objective is no longer describing your users’ experience.
Reduce n, reduce the per-call tail by a factor of ten per factor of ten in width, or stop waiting for every answer. If none is available, the remaining move is to hedge, which buys tail without buying service.
A hundred shards, each meeting a p99 objective, and a request that needs all hundred answers.
The third option is the one that gets used in practice, and it is worth seeing why it is not a cheat. Returning with ninety-five of a hundred shard results changes the event whose probability is being computed: the request no longer needs a conjunction of a hundred statements, so there is no pⁿ to compute. The cost is an answer that is slightly incomplete, and the whole design question is whether anyone can tell.
It is also worth noticing how little the second option buys per unit of effort. Moving every call from a p99 to a p99.9 is an order-of-magnitude reduction in the tail, and at n = 100 it takes the slow-request rate from 63.40% to 9.52% — a real improvement, and still not a number anyone would put on a slide.
For a faded rehearsal, take a fan-out of 40 with a per-call p95. The supplied first move is the break-even for a p95, which the lab prints as n = 14. Say whether 40 is past it, compute 0.95⁴⁰ and the slow-request probability, then say which of the lab’s four per-call percentiles brings that probability under ten percent at the same width. Check each figure in the lab before opening the quiz.
You should be able to compute pⁿ and 1 − pⁿ for a described fan-out, locate the break-even n for a given per-call percentile, and say which of the three levers — width, per-call tail, or not waiting for everything — a particular system has available.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.