A timeout and a retry count are a pair of numbers with two consequences each. The worst case is (r + 1) × T and it has to fit inside the allocation the budget gave this call. And the timeout itself discards every call that would have succeeded after it — 1 − F(T) of them — so a timeout set at the mean throws away fifteen percent of perfectly good calls on the lab’s service.
Core ideas
The hard idea is that a timeout is not a safety device with no cost. It is a decision to abandon calls that were going to work, and the size of that decision is a number you can read off the distribution.
A call with r retries can consume r + 1 full timeouts before giving up. That is the number the budget has to accommodate, not the typical case: on the lab’s preset, T = 90 ms with one retry is 180 ms against a 250 ms allocation, which fits, while a second retry would need 270 ms and does not.
Every call that would have finished after T is abandoned even though it was going to succeed. On the preset, T = 90 ms kills 1% — the one call in a hundred that takes 400 ms. Move the timeout to the mean of 18.27 ms and it kills 15%, which is fifteen working calls in every hundred turned into errors by a configuration value.
If attempts are independent, all r + 1 of them time out with probability (1 − F(T))ʳ⁺¹: at T = 90 ms that is 1/100 with no retry and 1/10 000 with one. The retry buys two decimal places of reliability and costs a doubling of the worst case, and the budget is what decides whether the second half of that trade is affordable.
The material
What the budget holds, what the timeout discards, and why the mean is the worst place to put it.
The lab’s preset is a hundred measured calls: forty-five at 8 ms, twenty-five at 11 ms, fifteen at 16 ms, eight at 24 ms, four at 45 ms, two at 90 ms and one at 400 ms. Its median is 11 ms, its p95 is 45 ms, its p99 is 90 ms and its mean is 18.27 ms. The budget has given this call 250 ms.
For a timeout T and a retry count r, the worst-case latency of the call is (r + 1) × T: every attempt may run to the timeout before the next begins. The kill fraction is 1 − F(T), where F is the measured distribution of call latency — the proportion of calls that would have succeeded and are abandoned. The call fails entirely with probability (1 − F(T))ʳ⁺¹ if attempts are independent.
Read the 250 ms budget across that table and the feasible choices appear immediately. T = 90 ms with one retry costs 180 ms in the worst case, kills 1% of good calls and loses the call entirely once in ten thousand. T = 90 ms with two retries needs 270 ms and does not fit. T = 125 ms with one retry uses the whole 250 ms exactly and kills the same 1%, because there is nothing in the distribution between 90 ms and 400 ms for the extra 35 ms to rescue.
It is a common instinct: the calls take 18.27 ms on average, so anything much past that has clearly gone wrong. On this distribution that setting kills 15% of good calls. Fifteen in every hundred requests that were going to return a correct answer are turned into errors, retried, counted as failures, and — if the retry goes to the same overloaded place — used to generate more load.
The reason is the shape from “Percentiles from a Sample”. The mean of a latency distribution sits well above the median and well below the tail, in the sparse region between the bulk and the outliers. A timeout there is close enough to the bulk to catch a great deal of ordinary variation and far enough from the tail to be no protection against the calls that are genuinely stuck. It is the worst of both, and it is the most common setting there is.
At T = 90 ms a single attempt is abandoned 1% of the time. With one retry, and assuming the two attempts are independent, both are abandoned 1/100 × 1/100 = 1/10 000 of the time — the call now fails once in ten thousand instead of once in a hundred. The cost is that the worst case doubles from 90 ms to 180 ms, and the budget has to hold the worst case rather than the typical one. Two decimal places of reliability for a doubling of the reservation is usually a good trade; a third retry, at 270 ms, is one this budget cannot make.
The independence assumption in that example deserves the same scepticism it got in “Hedged Requests”. If the call timed out because the far side is overloaded, the retry is a second request to an overloaded service and its probability of timing out is not 1/100; it is close to 1. Retries against a saturated dependency are not a reliability mechanism, they are a load multiplier, which is why the retry count is capped and why the cap matters more than its value.
Two figures should therefore be quoted together whenever a timeout is proposed: the worst case (r + 1) × T against the allocation, and the kill fraction 1 − F(T) against the measured distribution. A timeout chosen without the second is a number chosen without knowing what it throws away, and on this sample the difference between the p99 and the mean is a factor of fifteen in exactly that quantity.
Interactive
The worst case against the budget, and the good calls the cut-off discards
Method
Both numbers at once, because each one constrains the other through the budget.
It is the figure from “Latency Budgets”, and it is what the worst case has to fit inside. Without it there is no constraint and any timeout looks defensible.
Not a guess and not a mean: the actual proportion of calls that finish after T. Try the median, the p95 and the p99 and write down the kill fraction for each. The three numbers usually settle the question by themselves.
Divide the allocation by the timeout and round down; that is the largest number of attempts available, and one less than it is the retry count. If the answer is zero attempts, the timeout is too large for the budget and one of the two has to move.
A timeout proposal with only one of those numbers is incomplete. Add the failure probability (1 − F(T))ʳ⁺¹ if the attempts are plausibly independent, and say so explicitly if they are not.
Three candidate timeouts, the same distribution, and the budget doing the deciding.
The budget is what eliminated the third retry, and it did so without any judgement about reliability at all: 270 ms does not fit inside 250 ms. That is the value of having done the budget first. Without it the retry count is chosen by taste, and taste has no way of knowing that the third attempt would break an end-to-end promise made three lessons ago.
Notice the T = 125 ms row in the lab, which uses the entire 250 ms on two attempts and kills exactly the same 1% as T = 90 ms. The extra 35 ms of patience rescues nothing, because the distribution has nothing in it between 90 ms and 400 ms. Timeouts should be placed against the gaps in the measured distribution, and the gaps are visible only if the distribution is looked at.
For a faded rehearsal, keep the sample and set the allocation to 150 ms. The supplied first move is the attempt count at the p99 timeout: ⌊150 / 90⌋ = 1, so no retry is available there. Work out whether T = 45 ms with one retry fits, what it kills, and what its failure probability is, then decide between the two and write down the two figures you would quote. Check them on the lab’s sliders before opening the quiz.
You should be able to compute (r + 1) × T against an allocation, read 1 − F(T) off a measured distribution for several candidate timeouts, choose the largest retry count the budget permits, and state the failure probability together with the assumption it rests on.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.