A fleet percentile is the percentile of the merged sample: one definition, applied once, to every request. The number most dashboards show instead is the mean of the per-host percentiles, which is not a percentile of anything and is not an estimate of one. On the sample in the lab the three host p99s are 12, 14 and 400 ms, their mean is 142 ms, and the fleet median is 180 ms — so the average of three 99th percentiles lands below the middle of the distribution it is supposed to sit far above.
Core ideas
The definition is the whole lesson. What has to be added to it is why the wrong number looks so reasonable, and what it is actually weighting.
Nearest rank, as “Percentiles from a Sample” on the Latency and the Tail course defines it: sort the n values, take the one at position ⌈q·n⌉. Applied to one host’s requests it gives that host’s p99; applied to every request the fleet served it gives the fleet’s. These are two different samples, so they are two different questions, and only the second one is about the fleet.
That is the entire mechanism. In the lab’s sample one host served 90 of the 100 requests and the other two served five each, so averaging the three p99s gives the busy host one vote out of three when it deserves ninety out of a hundred. The result, 142 ms, is dragged down by two hosts that between them handled a tenth of the traffic.
If every host reports a p99 at or below v, then at least 99% of each host’s requests are at or below v, so at least 99% of all of them are, and the fleet p99 is at or below v as well. The same argument downward gives min pᵢ ≤ fleet p ≤ max pᵢ, and the mean is inside that range too. So the wrong number is never absurd, which is precisely why it survives review. It is simply not the answer to any question.
The material
One definition applied to two samples, why the average of per-host percentiles is not an estimate of the fleet value, and what the per-host values do and do not determine.
Given the requests served by every host in a fleet over a window, the fleet q-th percentile is the q-th percentile of that single merged sample: sort all n latencies and take the one at position ⌈q·n⌉. It is computed from requests, not from per-host summaries, and every request counts once.
There is no second definition. The bucketed percentile of “Histograms and Bucket Error” and the corrected percentile of “Coordinated Omission” are this same rule applied to a coarsened sample and to a repaired one. Where two of this course’s numbers disagree, the disagreement is about which requests were in the sample, never about what a percentile is.
The number a metrics system usually makes easy, by contrast, is one p99 per host per minute, and the obvious thing to do with several of those is to average them. That average has a name in no textbook. It is not the fleet p99, it is not an estimate of the fleet p99, and it does not converge to the fleet p99 as the window grows.
Read the last three lines together. The average of three 99th percentiles is 142 ms; the fleet’s median is 180 ms. The number being presented as a statement about the slowest one per cent of requests is smaller than the number that half of all requests beat. It is not a slightly optimistic p99. It is below the middle.
Let host i have nᵢ requests and q-th percentile pᵢ, and let M = max pᵢ. At least a fraction q of each host’s requests are at most pᵢ ≤ M, so at least a fraction q of all Σnᵢ requests are at most M, and the merged q-th percentile is therefore at most M. Running the argument with m = min pᵢ from the other side gives
min pᵢ ≤ fleet p ≤ max pᵢ.
One corollary is worth stating because the folklore gets it backwards: if every host reports the same p99, the fleet p99 is that number exactly — the bound closes. Percentiles fail to average, but they do not fail by leaving the range of the values being averaged.
So the mean of the per-host values is always somewhere plausible, and it is why nobody notices. What the bound does not give you is the value: anywhere inside [min, max] is consistent with the per-host figures, and where the fleet value actually falls depends on data the per-host summaries threw away.
That claim deserves a demonstration rather than an assertion, so here are two fleets. Both have two hosts, both hosts serve a hundred requests, and in both fleets the two host p99s are 40 ms and 100 ms. The fleet p99 is not the same number in the two cases.
Fleet one. Host X is 98 requests at 10 ms, one at 40, one at 50; its p99 is position 99 of 100, which is 40 ms. Host Y is 98 at 20 ms, one at 100, one at 200; its p99 is 100 ms. Merged, n = 200 and the fleet p99 is position 198, which is 50 ms.
Fleet two. Host X is 98 at 10 ms and two at 40; its p99 is again 40 ms. Host Y is 96 at 30 ms and four at 100; its p99 is again 100 ms. Merged, position 198 of 200 is 100 ms.
Same two host p99s, fleet answers of 50 ms and 100 ms. The average, 70 ms, is a fact about neither fleet, and it is not even between the two right answers in any useful sense: it is above one and below the other. Both fleet values sit inside [40, 100], exactly as the bound promises, and the bound is the most that can be said without the requests.
Counts and sums combine, because they are additive: total requests, total errors, total milliseconds. A ratio of two sums is fine — the fleet error rate really is the sum of the errors over the sum of the requests — and so is a mean weighted by its own denominator. It is the order statistics that do not: a percentile, a median, a maximum-of-percentiles all need the sample.
The practical answer is to keep something mergeable. A histogram of counts per bucket adds bucket by bucket, and the percentile is read off the merged histogram rather than averaged across hosts; that is what almost every metrics system is doing when it gets this right. It costs an interval instead of a point, which is the subject of the next page, and an interval that contains the answer is a better object than a point that does not.
Interactive
Two answers from one definition and one set of requests
Method
The first step is the one that gets skipped, and skipping it is what makes the rest impossible.
If the answer is “several samples”, it is not a percentile yet. Write down how many requests each contributing host served; the moment those counts are unequal, the mean of the per-host values is weighting hosts rather than requests.
Sort the combined latencies and take position ⌈q·n⌉. This is the answer. Everything else on this page is about what to do when you cannot get the requests.
Bucket counts add across hosts, so the merged histogram is exact even though the percentile read from it is an interval. An interval that certainly contains the fleet p99 beats a point that certainly is not it.
With only per-host percentiles, min pᵢ ≤ fleet p ≤ max pᵢ is everything that can honestly be said. Report the range. If the range is too wide to act on, that is a finding about the pipeline, not about the service.
Most panels that plot one line per host and then add an aggregation are averaging percentiles by default. Find the aggregation setting, read it, and write the answer in the panel description so the next person does not have to.
The three samples are the lab’s preset. Two of the hosts are healthy and lightly loaded; the third is carrying the traffic and is slow. Nothing here is adversarial — it is what a fleet behind a load balancer with a stuck host looks like.
The bound holds throughout: 12 ≤ 400 ≤ 400, and the average 142 is inside [12, 400] as well. Nothing here is out of range. The average is wrong by being an answer to no question, not by being extreme, and that is why it needs a worked example rather than a warning.
Notice also what a per-host p99 means on a five-request host: nearest rank at ⌈0.99 × 5⌉ = 5 is the maximum. Host A’s “p99” of 12 ms is its slowest request out of five, and two of the three numbers being averaged are maxima of tiny samples. The lab makes this visible — move requests onto A and watch its reported p99 stop being its maximum.
For faded practice, keep the three samples and change only the division of traffic: give host C thirty requests and hosts A and B thirty-five each, drawn from the same value lists. The supplied first move is that the three per-host p99s barely change. Predict the new fleet p99 and the new average before you compute either, then check both in the lab and say which of the two moved and why.
You should be able to compute a fleet percentile from merged samples by nearest rank, compute the mean of the per-host values on the same data and say what it is, state and use the bound min pᵢ ≤ fleet p ≤ max pᵢ, and name what your own pipeline aggregates.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.