A cache is described by its hit rate and judged by its miss rate. The backend never sees h; it sees 1 − h, multiplied by the arrival rate. So the interesting number is the one nobody quotes, and moving a cache from ninety to ninety-nine per cent is a tenfold cut in what arrives rather than a nine-point improvement.
Core ideas
One multiplication, and two habits of reading it. The multiplication is easy; the habits are what make the answer feel wrong the first time.
Requests arrive at λ. A fraction h is served from the cache and never leaves it; the rest, (1 − h)λ, is forwarded. At λ = 10 000 requests a second and h = 90% the cache absorbs 9 000 and the backend receives 1 000. Every sentence about the backend — its load, its utilisation, its wait, its bill — is a sentence about 1 − h.
From 90% to 95% the miss rate goes 1/10 → 1/20, so the backend load is halved. From 90% to 99% it goes 1/10 → 1/100, so the load is divided by ten. Five points and nine points look comparable written down and are not: what matters is the ratio of the two small numbers, not the gap between the two large ones.
Backend load divided by backend capacity is the utilisation of Queues and Utilisation. At μ = 2 000 requests a second, 1 000 in means ρ = 1/2 and 100 in means ρ = 1/20. A cache is a utilisation control, and the wait it buys is not proportional to the load it removes.
The material
Why the same improvement reads as small on one side of the cache and enormous on the other.
For a stream of requests arriving at rate λ, the hit rate h is the fraction served from the cache and the miss rate 1 − h is the fraction forwarded. The backend load is (1 − h)λ, in the same units as λ.
The two rates are one number written two ways. Which one you write down decides what the sentence is about: h is about the cache, 1 − h is about everything behind it.
There is no model in that definition and no assumption to argue with. It is a split of a rate into two parts, and it is exact whenever the hit rate is measured over the same window as the arrival rate. What it is not is stable: h is a property of the workload and the cache together, and a change in either moves it.
λ = 10 000 rps and h = 90%. The cache absorbs (9/10)(10 000) = 9 000 rps and the backend receives (1/10)(10 000) = 1 000 rps. Against a backend that can serve μ = 2 000 rps that is ρ = 1 000/2 000 = 1/2, so the backend is busy half the time.
Take the same λ and improve the cache. At h = 95% the backend receives (1/20)(10 000) = 500 rps: the hit rate went up by five points and the backend’s work halved. At h = 99% it receives 100 rps, a tenth of what it received at 90%. Nothing about the backend improved by nine per cent.
The λ cancels, which is why the factor is a property of the two hit rates alone. It is also why the same percentage point is worth wildly different amounts at different places on the scale: 50% to 51% divides the backend load by 50/49, about two per cent, while 98% to 99% divides it by two.
At λ = 10 000 rps into a backend of μ = 2 000 rps: h = 90% leaves 1 000 rps and ρ = 1/2; h = 95% leaves 500 rps and ρ = 1/4; h = 99% leaves 100 rps and ρ = 1/20; h = 99.9% leaves 10 rps and ρ = 1/200. The hit rates in that list are nearly the same number. The utilisations differ by a factor of a hundred.
Backend load on its own does not tell you what the backend feels. “The M/M/1 Queue” gives the mean wait at a rational utilisation as W = 1/(μ − λ), and on these four rows that is 1.00 ms, 0.67 ms, 0.53 ms and 0.50 ms. Notice how little the last two moved: past the knee, further hit rate buys almost nothing, because the queue was already nearly empty. “The Knee: Response Time vs Utilisation” is the curve that says which side of it you are on.
So a cache is worth the most exactly where the backend is worst. Going from ρ = 0.95 to ρ = 0.5 is the difference between a system that falls over and one that does not; going from ρ = 0.05 to ρ = 0.005 is the difference between two numbers nobody can feel. The hit rate is the lever and the utilisation is the load, and it is the load you should be looking at.
Two cautions before the lab. First, (1 − h)λ is the average backend rate, and the backend’s bad moments are not averages — a hot key expiring sends a burst through that this formula does not see, which is what Cache Stampedes is for. Second, the hit rate is measured, not chosen: writing 99% into a capacity plan is a forecast about a workload, and it deserves the error bar Capacity Estimation taught you to carry.
Interactive
(1 − h)λ is what arrives there, and ρ is what it does
Method
Four lines, and the first one is the whole discipline: write down the miss rate before anything else.
h = 99% becomes 1 − h = 1/100. Do it as a fraction rather than a decimal: 1/100 invites you to compare it with 1/10, and 0.01 invites you to compare it with 0.10, which reads as a small difference.
(1 − h)λ is the backend load in the units λ came in. Keep the units visible — requests per second, not just a number — because the next step divides it by a capacity that must be in the same units.
ρ = (1 − h)λ/μ is what the backend is actually running at. If ρ ≥ 1 the cache has not saved it: the backlog grows without bound and there is no mean wait to quote.
Two hit rates give a factor (1 − h₂)/(1 − h₁), and that factor is the honest way to report a cache improvement. “We halved the database load” is a claim someone can check; “we improved the hit rate by five per cent” is a claim about the wrong side of the cache.
The arithmetic is one multiplication and one division. What is worth watching is how the four columns move at different speeds.
Read the third column and the fifth column together. Between 90% and 99% the backend load falls by a factor of ten and the mean wait falls by about half: the load is linear in the miss rate and the wait is not. That is the whole reason to compute both.
For a faded rehearsal, take λ = 40 000 rps into a backend of μ = 1 000 rps. The supplied first move is that the backend is overloaded at h = 95%: check that, then find the smallest hit rate in tenths of a per cent that puts ρ below 1/2, and state the factor by which the load fell relative to 95%. Run the lab at your answer before opening the quiz.
The lab recomputes every row from (1 − h)λ as you move the sliders, and prints the backend’s wait from the M/M/1 result of Queues and Utilisation beside it. Drag the hit rate from 90% to 99% and watch which of the four numbers actually moves.
You should be able to convert a hit rate and an arrival rate into backend requests per second, divide by a capacity to get ρ, and state a cache improvement as the factor by which the backend load fell rather than as a change in percentage points.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.