A percentile is a position in a sorted list, not an average: the p99 is the ⌈0.99n⌉-th value, which means it is a time some request actually took. The mean is a different kind of number about a different question, and on a latency sample it is usually a value no request came close to. Computing both on the same twenty measurements is what makes the difference stop being a slogan.
Core ideas
One idea, and it is a definition: p99 names a position in a sorted list. Everything that makes percentiles behave strangely later follows from that and not from anything statistical.
Sort the sample. Compute r = ⌈q · n⌉. Take the r-th value. For q = 0.99 and n = 20 that is ⌈19.8⌉ = 20, so the p99 is the twentieth of twenty values. No arithmetic is done on the numbers themselves — they are only compared — which is why the answer is always a measurement rather than a construction.
The mean of this sample is 53.2 ms, and 17 of the 20 requests were faster than that. It is not a typical experience and it is not meant to be: the mean times the count is the total time spent, which is the right number for capacity and cost and the wrong number for describing what a user saw.
With n = 20 the rank for q = 0.99 is 20, so the p99 of this sample is its maximum — one measurement, carrying the whole claim. A percentile is only as trustworthy as the number of observations beyond it, and 1/n is the finest resolution a sample of size n has.
The material
The definition, then the same twenty numbers read four different ways.
For a sample of n values sorted ascending and a fraction q between 0 and 1, the q-th percentile by nearest rank is the value at position r = ⌈q · n⌉. It is one of the observed values, always. Other definitions exist that interpolate between neighbours and can return a number nobody measured; this course uses nearest rank throughout and prints the rank beside the value.
The lab’s preset is twenty measurements in milliseconds, already sorted: 12, 13, 14, 14, 15, 15, 16, 17, 18, 19, 21, 22, 24, 27, 31, 38, 52, 96, 180, 420. The ranks are the whole computation.
Four numbers describing one sample: 19, 96, 180, 420. The median is 19 ms and the p99 is twenty-two times that. A single summary figure cannot carry that spread, and choosing which single figure to carry is the decision this lesson is about.
The mean is 53.2 ms. Seventeen of the twenty requests were faster than the mean, and no request in the sample took anything near 53 ms — the values on either side of it are 52 and 96. The mean has landed in a gap. This is not a defect of this sample; it is what happens to an average whenever a few large values are pulled far out, which is the normal shape of a latency distribution.
No user saw 53 ms. The claim, read as a claim about a typical experience, is false in this sample by construction: 85% of the requests were faster and 15% were much slower, and the number in the middle belongs to neither group. What is true of the mean is that twenty requests consumed 20 × 53.2 = 1064 ms of service time in total — a fact about load, which is exactly what a capacity calculation needs and exactly what a latency report does not.
Notice what removing one measurement does. Delete the 420 from the sample in the lab and the p99 of the remaining nineteen becomes 180 ms, less than half of what it was, because at n = 19 the rank ⌈0.99 × 19⌉ = 19 now selects the new maximum. A high percentile of a small sample is a claim resting on one or two observations, and moving one of them moves the answer by a factor of two. The fix is more measurements, not a different formula.
The rank definition also says something about how the value is found. Sorting is the obvious method and it is what the lab shows, because seeing the sorted list is the point. It is not the only method: “Quickselect” on the Algorithms path finds the r-th smallest value without sorting the rest, which is how a system with millions of measurements a second computes the same number. The definition does not change; only the work does.
Interactive
Four numbers about one sample, three of which a request actually took
Method
Do this by hand once. It is four steps and it permanently changes how the word p99 reads.
Every step after this is a position in that list. If the sample arrived as a stream of numbers this is the only part that costs anything, and it is also the part that makes the rest obviously correct.
Round up, always: ⌈0.99 × 20⌉ = 20 and ⌈0.99 × 200⌉ = 198. Rounding down or to nearest gives a neighbouring value, which on a tail can be very different from the right one.
It is a time some request actually took. Say which one it is if the sample is small — “the p99 is the slowest of twenty” is a more honest sentence than “the p99 is 420 ms” when both are the same fact.
The mean times the count is the total service time, which is what capacity and cost are made of. Do not report it as an experience, and check how many observations fall below it — on a latency sample the answer is usually most of them.
The sample is small enough to count on your fingers, which is the point: everything here is a position, and the positions can be checked.
The 22.1× ratio at the bottom is the shape of the distribution in one number, and it is the figure that makes the rest of this course necessary. A system whose p99 is twenty-two times its median does not have a latency problem in any stage; it has a tail, and the tail is a separate object with its own arithmetic.
It is worth noticing that the p50 and the mean disagree by nearly a factor of three, in the direction they always disagree: the mean is above the median whenever the large values are the far ones. If a system reports a mean close to its median, either the tail is unusually tame or something is truncating the measurements — which is its own subject, and belongs to Measuring Systems.
For a faded rehearsal, take the eleven measurements 5, 6, 6, 7, 8, 9, 11, 14, 20, 60, 300. The supplied first move is the size: n = 11. Compute the ranks and the values for p50, p90 and p99, then the mean, then count how many observations are below the mean. Say which single observation the p99 rests on, and type the sample into the lab to check every figure before opening the quiz.
You should be able to compute a rank, select the value it names, compute the mean of the same sample, count how many observations fall below it, and state how many observations the reported percentile actually rests on.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.