Under a cache a request has two possible latencies and nothing in between. The mean is the weighted average of the two, with the hit rate as the weight, and it is a number no request ever takes. The percentile is a different object: a rank on the same two outcomes, so it is always one of them, and it does not move gradually — it steps.
Core ideas
The distribution here has exactly two values in it. That is what makes both calculations short and what makes them disagree so visibly.
E[latency] = h·t_hit + (1 − h)·t_miss is the expected value of Discrete Mathematics applied to a two-outcome random variable. At h = 95%, t_hit = 1 ms and t_miss = 40 ms it is 59/20 = 2.95 ms — nowhere near the midpoint 20.5 ms, because the outcomes are not equally likely.
Every request takes 1 ms or 40 ms. 2.95 ms is a property of the stream, not of any request in it, and it is the number that a cache improvement moves smoothly. If somebody reports a mean latency of 3 ms for a cached endpoint, one request in twenty is still taking 40.
P(latency ≤ t_hit) = h and P(latency ≤ t_miss) = 1. There is nothing between the two values, so the q-quantile is t_hit exactly when h ≥ q and t_miss otherwise. It cannot land between them, it does not interpolate, and it jumps by t_miss − t_hit all at once at h = q.
The material
The mean, the staircase it comes from, and the one hit rate at which each percentile flips.
A request is served in t_hit with probability h and in t_miss with probability 1 − h. Its mean latency is E[latency] = h·t_hit + (1 − h)·t_miss, and its distribution takes only those two values.
t_miss here is the whole miss path: the lookup that failed, the backend call, and the write back into the cache. It is not the backend’s service time alone.
The mean lies strictly between the two latencies whenever h is strictly between 0 and 1, and it equals neither. It sits close to t_hit because h is close to 1: the weighted average is dragged toward whichever outcome is common, and in a cache that is the hit.
h = 95%, t_hit = 1 ms, t_miss = 40 ms. The mean is (19/20)(1) + (1/20)(40) = 19/20 + 2 = 59/20 = 2.95 ms. The unweighted average of the two latencies is 20.5 ms, seven times too large, and it is the answer you get by forgetting that one outcome happens nineteen times as often as the other.
A percentile asks for a value, not an average: the q-quantile is the smallest t with P(latency ≤ t) ≥ q, which is the nearest-rank convention of “Percentiles from a Sample”. Here the function P(latency ≤ t) is a staircase with two steps — it is 0 below t_hit, h from t_hit up to t_miss, and 1 at t_miss and above.
At h = 95% that gives p50 = 1 ms, p95 = 1 ms, p99 = 40 ms and p99.9 = 40 ms. The median and the p95 are the hit latency; the p99 is the miss latency, because more than one request in a hundred misses and the hundredth-slowest is one of them.
The q-quantile becomes t_hit at exactly h = q and not before. So the p99 is the miss latency for every hit rate below 99%, and the p99.9 is the miss latency for every hit rate below 99.9%. Improving a cache from 95% to 98.9% does not move the p99 at all, and then one more tenth of a point moves it by 39 ms.
t_hit = 1 ms, t_miss = 40 ms. At h = 95% the mean is 2.95 ms and the p99 is 40 ms. At h = 99% the mean is 139/100 = 1.39 ms and the p99 has just become 1 ms — but the p99.9 is still 40 ms. At h = 99.9% the mean is 1039/1000 = 1.039 ms and the p99.9 is 1 ms too. The mean moved by under two milliseconds across that whole range; the tail moved in two jumps of thirty-nine.
This is why a cache is a poor answer to a tail problem and a good answer to a throughput problem. It removes work, which is what Hit Rate and Backend Load measured, and it shifts the mean, but the slow requests it leaves behind are exactly as slow as they were. To move the tail you have to move t_miss, or push h past the percentile you care about — and the second of those gets expensive fast, which is what Cache Size and Hit Rate is about.
The model assumes two latencies and one hit rate. A real cache has a spread on each — a hit is 0.8 to 1.4 ms, a miss depends on what the backend was doing — so treat t_hit and t_miss as the typical values and read the answer as a shape rather than a measurement. What survives the simplification is the staircase: two clusters with a gap, and a percentile that lands in one cluster or the other.
Interactive
An expectation over two outcomes, and the percentile that ignores it
Method
Compute the mean first because it is one line, then stop trusting it and compute the rank.
t_hit, t_miss and h, with t_miss covering the whole miss path including the write back. Getting t_miss wrong is the largest error available here, because it carries almost all of the tail.
h·t_hit + (1 − h)·t_miss. Keep h as a fraction and the answer stays exact: (19/20)(1) + (1/20)(40) = 59/20. Sanity-check it against t_hit: a mean far above the hit latency means h is lower than you thought.
No arithmetic is needed. h ≥ q gives t_hit; otherwise t_miss. Do this for every percentile anyone quotes to you, because the answer is different for each and there is no interpolation to fall back on.
A cache improvement moves the mean smoothly and moves each percentile once, at one specific hit rate. Reporting “latency improved” without saying which of the two you mean is how a team ships a change that helps the average and leaves every slow request exactly as slow.
One expectation and four comparisons. The comparisons are the part worth slowing down for, because each one has a different answer.
The p95 line is the one to look at twice. h = q exactly, and the rule is ≥, so the p95 is the hit latency — the boundary belongs to the hit. One tenth of a point lower and it is 40 ms.
For a faded rehearsal, take t_hit = 2 ms, t_miss = 120 ms and h = 99%. The supplied first move is that the mean is (99/100)(2) + (1/100)(120); finish it, then find the smallest hit rate in tenths of a per cent at which the p99.9 becomes the hit latency, and say what the mean is there. State which percentile you have not moved.
In the lab the staircase is drawn with the four percentile lines across it. Drag the hit rate slowly from 95% upward and watch the mean slide while the p99 sits still, then jumps.
You should be able to produce the mean latency as a weighted average, decide any percentile by comparing h with q, and say which percentiles a proposed hit-rate improvement will and will not move.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.