W = S/(1 − ρ) is the single most useful rearrangement on this path. Divided through by S it says the response time is the reciprocal of the headroom: two service times at 50%, ten at 90%, a hundred at 99%. It is a hyperbola, its asymptote is at ρ = 1, and nothing a designer can buy moves it.
Core ideas
There is one function here and two ways of asking questions of it. The second way is the one capacity planning needs.
W/S = 1/(1 − ρ) has 1 − ρ in the denominator, so what determines the wait is how much of the server is spare. Halving the spare capacity doubles the response time regardless of where you started, which is why the same ten percentage points of load cost five times as much at the top of the range as at the bottom.
As ρ approaches one the curve rises without bound. No faster disk, no better garbage collector and no amount of tuning moves a vertical asymptote: those things change S, which scales the curve, while the blow-up stays exactly where it was. Algebra's graphs-and-asymptotes is the shape.
Nobody picks a utilisation and accepts the response time that falls out. The useful direction is to name what you will tolerate — W = kS — and solve for ρ = (k − 1)/k. Tolerate four service times and you may run at three quarters; tolerate ten and you may run at nine tenths.
The material
The curve, three points on it, the inverse question, and what the last tenth of a machine actually costs.
From the previous lesson, W = L/λ with L = ρ/(1 − ρ) gives W = 1/(μ − λ), and dividing numerator and denominator by μ turns that into W = S/(1 − ρ) with S = 1/μ. It is the same quantity in the units a designer thinks in: multiples of how long the work itself takes.
W/S = 1/(1 − ρ) is how many service times a request spends in the system. It is 1 at ρ = 0 — the work and nothing else — and it is unbounded as ρ approaches 1. The knee is not a particular point on this curve; it is the fact that the curve has no particular point, and that the interesting region is always the last few percent.
Notice what those two multipliers have in common: the first covers forty points of utilisation and the second covers nine, and the second is the more expensive. The utilisation axis is the wrong axis to reason on. The headroom axis — 1 − ρ, going 1/2, 1/10, 1/100 — is the one the response time is actually a function of.
Set W = kS and solve. 1/(1 − ρ) = k gives ρ = (k − 1)/k, so the headroom you must keep is exactly 1/k. Tolerating twice the service time means running at 1/2; four times means 3/4; ten times means 9/10; a hundred times means 99/100, which is the point at which a single machine failure in a pool is an outage rather than an event.
Capacity Estimation divided an offered rate by a target utilisation rather than by μ, and took the target on trust. This is where it is earned: the target is (k − 1)/k for whatever k multiple of the service time the design is willing to pay. A plan that names 80% has chosen k = 5, whether or not anybody said so.
The curve also explains a familiar and confusing incident shape. A service at ρ = 1/2 absorbs a doubling of traffic and moves from 2S to 4S, which nobody notices. The same doubling at ρ = 9/10 is not survivable at all: ρ would be 9/5, above one, and there is no response time to quote — only the growth rate of “λ, μ and ρ”. Two systems, the same relative traffic change, and entirely different outcomes.
Two honest caveats. The constant in W = S/(1 − ρ) is M/M/1's: the 1/(1 − ρ) shape is general and the multiplier in front of it depends on how variable the arrivals and the service times are, which the next lesson prices. And W is a mean; the percentiles that Latency and the Tail measured sit further out still, because the number in the system is geometrically distributed rather than concentrated around its average.
Interactive
Response time against utilisation, and the asymptote you cannot tune away
Method
Start from what you will tolerate, not from what the machine can take.
S is the service time with no queueing in it — the work itself. A measurement taken under load already contains the waiting this curve is about, and using it makes the arithmetic circular.
k in W = kS. This is a product decision expressed as arithmetic: two service times, four, ten. It is the only free parameter on the page.
ρ = (k − 1)/k, leaving 1/k of headroom. Quote the headroom alongside the utilisation, because the headroom is the quantity the response time depends on.
Multiply the offered load by whatever surge the system has to survive and recompute ρ. If it crosses one, the plan does not have a response time, it has a growth rate.
The same service, the same work, three loads — and then the question capacity planning actually asks.
The three headroom figures are the whole table restated: 1/k of a machine kept idle buys a response time of k service times. Nothing else on this page is doing any work.
It also shows why “we have 10% headroom” and “we have 1% headroom” are not neighbouring states. They differ by a factor of ten in response time, and the second one has no room for the variability of “Why Queues Form at ρ < 1” at all.
For a faded rehearsal, take S = 4 ms and a tolerance of W = 5S. The supplied first move is that k = 5 gives ρ = 4/5. Compute W at that target, then compute what W becomes if the service time degrades to 5 ms with the load unchanged.
You should be able to compute W/S at any rational ρ, invert it to a target utilisation and a headroom, explain why nine points of load cost more than forty, and say what a plausible surge does to the point you have chosen.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.