A machine’s capacity is one over what a request costs it, and the fleet is the peak divided by the capacity you are willing to use. Two things make this more than a division. The answer is a ceiling, because machines are integers and most of a machine will not serve anything. And the utilisation you divide by is a decision that has to be named: at 60% the fleet here is 167 machines, at 100% it is 100, and the difference is not a rounding.
Core ideas
One hard idea, the ceiling, with the two quantities it needs on either side of it: what a request costs, and how full you are willing to run.
A request that occupies one core for 40 ms lets a machine with 8 cores serve 8 · 1000/40 = 200 requests a second. The general form is the same for any resource: capacity is the reciprocal of what one request consumes, in whatever unit is scarce. Choosing which resource is scarce is the modelling decision; the arithmetic after that is a reciprocal.
20 000 requests a second over 120 usable gives 500/3 machines, which is 166.67 and therefore 167. Machines come in whole numbers, so the exact fraction is never the answer — and rounding down for tidiness gives a fleet that is short at exactly the moment the peak arrives. The lab draws the staircase so the 2/3 of a machine at the top of the last tread is visible.
60% is a decision about how full a queue should be allowed to run, and it comes from queueing theory rather than from uncertainty about the inputs. It is taken as given on this course. “The Knee: Response Time vs Utilisation” is where it is earned, and the short version is that response time rises steeply as utilisation approaches one, so the last 40% of a machine buys latency you cannot pay for any other way.
The material
The reciprocal that gives a machine’s capacity, the target that reduces it, and the ceiling that turns the quotient into a fleet.
The capacity of a machine is the reciprocal of the scarce resource one request consumes. Its target utilisation ρ is the fraction of that capacity the design is willing to use, so usable capacity is capacity · ρ. The machine count is N = ⌈peak / (capacity · ρ)⌉.
Every term has to be a peak or none of them can be. The rate going in is the peak rate from “Peak to Average”, not the mean; a fleet sized from a mean rate is short for as long as the busy period lasts, and no amount of care about the ceiling rescues it.
59.88% rather than 60% is the ceiling showing up in the answer: rounding 166.67 up to 167 bought two-thirds of a machine that nobody asked for, and the fleet is very slightly emptier than the target. That is the normal direction for this error to run, and it is the safe one.
Eight cores, 40 ms of one core per request, 60% target. Each machine serves 200 requests a second and is allowed 120, so the fleet is 167 machines carrying 33 400 requests a second of raw capacity, 13 400 of which is deliberately unused. Sized at 100% the same peak gives 100 machines, and the sixty-seven machines between those two answers are the headroom, priced.
Not insurance against the estimate being wrong. The target utilisation is a queueing decision: as utilisation approaches one, waiting time rises without bound, and a fleet run at 95% has a latency profile that no amount of extra patience fixes. 60% is a point on that curve, chosen so that the response time is tolerable and a machine can be lost without the rest going over the knee.
The uncertainty in the inputs is a separate and multiplicative thing. If the peak rate and the per-request cost are each known to a factor of two, the ranges from “Orders of Magnitude” put the fleet somewhere across a ÷4 … ×4 interval, and the headroom does not begin to cover that. Two different questions, two different answers, and conflating them produces a fleet that is neither adequately provisioned nor honestly estimated.
167 looks like a hard number and is only as hard as its inputs. Halve the per-request cost in the lab and the same peak and target give 84; double it and they give 334. Nothing about the ceiling makes 167 more certain than the 40 ms it came from — the ceiling removes the fractional part, not the range. The honest report names the fleet and the assumption that moves it most, which here is the cost of a request.
This is the one-constraint version: the fleet is sized by request throughput and by nothing else. Real fleets are often held up by a second constraint at the same time — a data volume that will not fit, a connection count, a per-node write rate — and then the answer is the larger of the two ceilings rather than either one. “How Many Shards”, on Partitioning and Load Balancing, is that lesson: the same ceiling computed twice and a maximum taken. Doing the one-constraint case first is worth it, because the trap there is always to compute one ceiling and stop.
Interactive
An integer, with a headroom you have to name
Method
The scarce resource first, then a reciprocal, then a target, then a ceiling.
CPU milliseconds, memory, disk I/O, connections — one of them runs out first and it is the one to count. Getting this wrong produces a fleet sized by a resource that was never scarce, and the symptom is a system that falls over at a fraction of its computed capacity.
Capacity is the reciprocal: cores × 1000 ms/s ÷ cost in ms for a CPU bound. Keep the units on the page — the 1000 ms/s row is the one people drop, and dropping it moves the answer by three orders of magnitude.
ρ = 0.6 is a decision, not a measurement, and it belongs in the write-up next to the answer. An estimate that reports N without reporting ρ cannot be checked by anyone, because the two numbers are only meaningful together.
⌈peak / usable⌉. Round up always: the fractional machine is doing real work at the peak, and the alternative is a fleet that is short for exactly as long as the busy hour lasts.
Say what N is, what ρ was, and what the per-request cost was — and then say what N would be if that cost were half or double. The integer looks exact and inherits every range that went into it.
Four inputs, one reciprocal, one ceiling, and a comparison against the answer that skips the headroom.
The two lines at the bottom are the ones that stop 167 from looking like a measurement. The per-request cost is a ranged factor like every other input on this course, and a factor of two in it is very nearly a factor of two in the fleet. The ceiling removed the fraction; it did not remove the range.
The 59.88% is worth noticing too. The target was 60% and the fleet runs just under it, because the ceiling handed you two-thirds of a machine you had not asked for. Over-provisioning by a fraction of a machine is the correct direction for that error, and it is the direction the ceiling always errs in.
For a faded rehearsal, take the lab’s edge-cache example: 500 000 requests a second, 2 ms a request, 4 cores, 80% target. The supplied first move is the capacity, 4 · 1000/2 = 2 000 requests a second per machine. Produce the usable capacity, the exact machine count and N, then predict what N becomes at a 60% target before changing the slider.
You should be able to turn a peak rate, a per-request cost and a target into N without assistance, say why the answer is a ceiling and never a rounding, distinguish the headroom from the uncertainty in the inputs, and name what N becomes if the per-request cost is out by a factor of two.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.