Above capacity a server spreads its work across more requests than it can finish inside a client's patience, so the share that completes is C/L and the goodput is C × (C/L) — it falls as the load rises. Refusing the excess at the door replaces that curve with a flat line at the threshold. At twice capacity, shedding delivers 900 requests a second against 500, so rejecting 55% of the traffic serves 1.8 times as many users as accepting all of it.
Core ideas
The collapse is not an assumption about pathological systems; it follows from one sentence about what happens to work spent on a request nobody is waiting for any more.
Above capacity the server still does C units of work a second, but it is spread across L requests and only the share C/L of them finish before the client gives up. The useful output is therefore C × (C/L) = C²/L, which falls as L rises. At twice capacity a thousand-per-second service delivers five hundred; at four times capacity, two hundred and fifty.
A shedder admits min(L, T) and puts that through exactly the same model. Below the threshold nothing changes; above it the admitted load is capped and the goodput is flat at T, provided T is at or below capacity. A threshold set above capacity collapses too, more slowly, which is why the threshold is chosen relative to the knee rather than to the load.
At twice capacity, refusing 55% of the traffic outright serves 900 requests a second where accepting everything serves 500. The rejected requests cost almost nothing — a fast error is cheap — and the accepted ones are the ones the server can finish. More users get an answer under the policy that says no.
The material
The model, both curves, where the threshold belongs, and what a circuit breaker is doing when it is the client that refuses.
Goodput is the rate of requests that are both served and still wanted. With capacity C and offered load L, this lesson's model is: for L ≤ C the goodput is L; for L > C the server's capacity is spread across L requests, only C/L of them complete within the client's patience, and the goodput is C × (C/L). A shedder with threshold T admits min(L, T) and passes it through the same model.
The model is this lesson's own and it is stated so that it can be argued with. The mechanism behind it — a queue that fills, work spent on entries whose clients have already left, and an admitted rate that is not the completed rate — is the subject of Queues and Utilisation's “Bounded Queues and Loss”, and the knee that makes the threshold choice non-obvious is “The Knee: Response Time vs Utilisation”.
Two features of the table decide everything. The shedder costs goodput at exactly one point — between the threshold and capacity, where it is refusing work the server could have done — and that cost is bounded by C − T, here a hundred requests a second. Everywhere above capacity it wins, and the margin grows without limit as the load rises.
The cost of a threshold below capacity is C − T at full load; the benefit is that the system never enters the collapsing regime. Since response time is already rising steeply as utilisation approaches one, the threshold is normally placed a little under capacity — 0.9C here — and the ten per cent is bought as insurance against the estimate of C being wrong.
Set the threshold to 120% of capacity and offer 200%. The shedder admits 1200 requests a second, which the server cannot serve, so the goodput is 1000²/1200 = 833.3 a second — better than the 500 of no shedding and worse than the 900 of a threshold at 90%. The shedder is not a magic boundary; it is a cap, and a cap above capacity caps nothing that matters.
With the shedder the utilisation is admitted/C = 0.9; without it, offered/C = 2.0. A utilisation above one is not a state a system occupies, it is a description of a queue growing without bound, and every second spent there adds latency that never comes back. The shedder's real product is that 0.9.
A circuit breaker is the same arithmetic with the client doing the refusing. After a threshold number of failures the client stops calling for a cooling period and fails fast locally, which removes its share of L at the source rather than at the server's door. The advantage over server-side shedding is that the rejected request never crosses the network; the disadvantage is that the client is deciding on stale local evidence about a service it has stopped observing.
Both mechanisms are about the same quantity and neither of them makes the offered load smaller. The requests that are shed still failed from the user's point of view — the argument is not that rejection is free, it is that rejection is cheap and collapse is not, so a fixed amount of failure buys far more successes when it is taken at the door.
At 200% offered load the unshed service succeeds on 500 of 2000 requests a second, a success ratio of 25%. The shed service succeeds on 900 of 2000, a ratio of 45%, while explicitly rejecting 1100. By the error-budget measurement of “Error Budgets”, which counts failed requests, the shedder spends less budget — 1100 rejections against 1500 timeouts a second — and the rejections are returned immediately rather than after a timeout.
Interactive
Goodput when you refuse work, against goodput when you accept all of it
Method
The first step is the hard one, because the threshold is expressed relative to a capacity that has to be measured rather than assumed.
Not the rate at which the service falls over: the knee, where response time begins rising steeply, is the capacity this model means. A figure taken from the maximum throughput ever observed is already inside the collapsing regime and will put the threshold in the wrong place.
T slightly under C — ninety per cent is a common choice. The cost is C − T of goodput at full load and the benefit is that an error in the capacity estimate, or a slow drift in request cost, does not push the system over the edge before the shedder engages.
min(L, T) through the model for the shedder, C²/L above capacity for the collapse. Compare them at the peak load you plan for and at two or three times it, because the ratio is what justifies the threshold and it grows with the overload.
A shed request must cost the server almost nothing and must be distinguishable by the client from a timeout, so that the client can back off rather than retry immediately. A rejection that is expensive to produce, or that looks like a failure worth retrying at once, feeds the loop rather than breaking it.
A service with a measured knee at 1200 requests a second meets a promotion that offers 3600. The question is what each policy delivers and what it costs in failed requests.
Shedding wins on both columns at once, which is the part that surprises people: it serves nearly three times as many users and produces fewer failures, because the failures it produces are cheap and the ones it avoids were consuming capacity on their way to failing anyway. The utilisation of 0.90 against 3.00 is the same fact stated as a queue property.
The cost is visible only in a narrow band. Between 1080 and 1200 requests a second the shedder refuses work the server could have done, up to 120 requests a second at the top of that band. That is the whole price of the policy, and it is paid in a region the system passes through rather than sits in.
For a faded rehearsal, the team proposes raising the threshold to 1200 — exactly capacity — to recover that band. The supplied first move is that the goodput at T = C is still C as long as the capacity estimate is exact. Work out what the goodput becomes if the true capacity turns out to be 1100 rather than 1200, under both thresholds, and say what that says about where the ten per cent was actually going.
You should be able to compute goodput under both policies, identify the narrow band where shedding costs something, place a threshold relative to a measured knee rather than to a maximum, and say why the shedding arm produces both more successes and fewer failures at high overload.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.