A run of n requests meets the slowest t fraction at least once with probability 1 − (1 − t)ⁿ. At the top 0.1% a six-hundred-request run manages 45.14% — worse than a coin toss — and ninety-five per cent confidence takes 2 995 requests. Even that only establishes that the tail was met once; measuring it takes an order of magnitude more again, and a percentile a run has fewer than one expected sample for is just the maximum wearing a label.
Core ideas
The same 1 − (1 − p)ᵏ as the sampling page, asked about requests instead of about events, plus the distinction that matters more than the formula.
Nearest rank returns the value at position ⌈q·n⌉, and that position always exists whatever n is. At n = 600 and q = 99.99% the rank is 600 — the largest observed value — so the tool reports a p99.99 and the number is the maximum of six hundred requests. Nothing in the output says so, and the more extreme the percentile you ask for, the more confidently the tool answers with the same number.
Each request is in the slowest t fraction with probability t and they are independent, so a run misses it entirely with probability (1 − t)ⁿ. At t = 1/1000, six hundred requests give 0.999⁶⁰⁰ = 54.86% of missing it. The break-even — the run length with an even chance — is n = 693, which is precisely the fan-out break-even of “Tail Amplification under Fan-out”, because it is the same equation with n moved from one side to the other.
A run with one sample in the top 0.1% can say that such a latency exists. It cannot say what the p99.9 is: that value is estimated by a single observation, and the next run will produce a different one. Putting ten samples above the percentile takes an expected 10/t = 10 000 requests, and a stable estimate takes more. The n on this page is a floor below which the number is meaningless, not the length that makes it good.
The material
Where the exponent comes from, why the answer is found by scanning rather than by a logarithm, and what confidence in a tail actually buys.
Let a request be in the slowest t fraction of the distribution with probability t, independently across requests. A run of n requests meets that tail if at least one of them falls in it, which happens with probability
1 − (1 − t)ⁿ.
For the p99.9, t = 1/1000; for the p99, t = 1/100; for the p99.99, t = 1/10000. Independence is the assumption, and it fails for a run that ramps, for a cache that is warming and for anything periodic.
A six-hundred-request run is a minute at ten requests a second, which is a completely ordinary thing to do, and it produces a report with a p99.9 in it. More than half the time that number was computed from a sample containing no request from the top 0.1% at all: rank ⌈0.999 × 600⌉ = 600 returns the maximum, and the maximum of six hundred draws from the body of the distribution is not the p99.9 of anything.
Solving (1 − t)ⁿ ≤ 1 − c for n is one logarithm, and the answer it gives here is n ≥ 2 994.23, so 2 995. The lab does not do that. It computes (999/1000)ⁿ as an exact fraction and compares it to 1/20 directly, increasing n until the comparison flips.
The reason is that the exact answer is a rational number a double cannot hold: (999/1000)²⁹⁹⁵ has a numerator of about nine thousand digits, and converting it to a floating-point number before comparing gives infinity divided by infinity. A scan over exact fractions has no such failure mode, and it also makes the boundary checkable — the lab can show that n = 2 994 does not clear the bar and n = 2 995 does, which a logarithm and a ceiling can only assert.
A one-minute run at 10 requests a second: 600 requests, p99.9 reported, and a 45.14% chance the run ever met the tail. A five-minute run: 3 000 requests, 95.03% — the number is now probably about something real, estimated from about three observations. A one-hour run: 36 000 requests, certainty of meeting the tail and roughly thirty-six samples above it, which is enough for the reported value to be stable between runs.
The three reports look identical. They differ by the length of the run, which is the one field nobody reads, and the first of the three is where most load-testing conclusions come from.
The expected number of requests above the q-th percentile in a run of n is n·t. One expected sample is the threshold at which the percentile stops being the maximum; it is nowhere near the threshold at which it is stable. As a working rule, ask for ten samples in the tail — n = 10/t, so 10 000 requests for a p99.9 and 100 000 for a p99.99 — and treat anything less as a lower bound with a wide error around it.
This is also the honest answer to “why did the p99.9 change between runs?” With n·t near one, it changed because it is the maximum of a random sample and maxima are volatile. Nothing regressed, nothing improved, and the difference between the two runs is the same kind of noise the first page of this course put an error bar on — except that a percentile deep in the tail has a far wider one than a ratio near the middle.
1 − (1 − t)ⁿ assumes every request is an independent draw from one fixed distribution. A run that ramps load over its first two minutes is drawing from a different distribution every second; a run against a cold cache is measuring the cache filling; a run short enough to sit inside one garbage-collection cycle may draw zero or all of its tail from that one pause. In each case the request count is right and the independence is not, and the practical response is to discard the ramp, warm the system first, and run for several periods of whatever the system does periodically.
Interactive
The probability a load test ever met the request it is reporting on
Method
Two of these are arithmetic and two are about what you do with the answer.
p99 is t = 1/100, p99.9 is 1/1000, p99.99 is 1/10000. If the report will quote several, size for the most extreme of them, because that is the one that will be quoted in the summary.
Under a half means the run is more likely than not to be reporting its own maximum. This is worth doing before the run rather than after, because afterwards there is a number on a slide and the argument becomes about the number.
n for 95% confidence is the floor. For a value you intend to compare against another run, size for about ten samples in the tail: n ≈ 10/t, which is 10 000 requests for a p99.9.
Discard the ramp, warm the caches, and make the run long enough to contain several of whatever cycle the system has — garbage collection, checkpoint flushes, a cron job. Independence is the assumption underneath every number above and it is the one a short run breaks.
Ten requests a second for a minute, which is what a quick check looks like. The question is not whether the service is fast; it is whether this run is in a position to say anything about the slowest one request in a thousand.
The 693 is the same integer as the fan-out break-even in “Tail Amplification under Fan-out”, and it is not a coincidence: that page asks how many parallel calls it takes for one of them to exceed its p99.9, this one asks how many sequential requests it takes to see one, and both are 0.999ⁿ = 1/2. The same arithmetic appears a third time on this course, in the false-alarm rate of a threshold alert over a month of windows.
Notice the last line. The gap between a run that is meaningless and a run that is useful is about seventeen minutes, which is less than the time usually spent arguing about the result of the meaningless one.
For faded practice, size a run that will report a p99.99 at 95% confidence. The supplied first move is t = 1/10000, so the required n will be about ten times the p99.9 answer. Predict it, find it in the lab by the scan, and then compute how long that run takes at a hundred requests a second — and whether ten samples in that tail is a run you would actually schedule.
You should be able to compute 1 − (1 − t)ⁿ for a run you are planning, find the n a stated confidence needs by an exact comparison, say how many more requests a stable estimate takes, and recognise a percentile that is the run’s maximum wearing a label.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.