A policy of r retries after a failed attempt costs (1 − pʳ⁺¹)/(1 − p) attempts per request and delivers success with probability 1 − pʳ⁺¹. Both are geometric series in the failure probability, and both move the wrong way together: as p rises the success falls and the cost climbs toward r + 1, so the policy is cheapest when it is least needed and most expensive at the moment the service can least afford it.
Core ideas
The same sum answers both questions a retry policy raises, which is why they cannot be traded against each other freely.
The first attempt always happens. The second happens with probability p, the third with p², and attempt i + 1 with pⁱ. Summing over the r + 1 attempts the policy allows gives the expected attempts per request: 1 + p + p² + … + pʳ = (1 − pʳ⁺¹)/(1 − p). That is the amplification factor — the number by which the offered load is multiplied before the service sees it.
A request is lost only when all r + 1 attempts fail, which under independence is pʳ⁺¹, so the policy succeeds with probability 1 − pʳ⁺¹. At p = 0.1 and r = 2 that is 99.9%, bought for 1.11 attempts a request — an eleven per cent surcharge for a factor of a hundred in the loss rate, which is why retries are standard.
(1 − pʳ⁺¹)/(1 − p) is increasing in p and tends to r + 1 as p approaches one. The same two-retry policy that costs 1.11 attempts at a 10% failure rate costs 2.71 at 90% and 2.97 at 99%. The surcharge arrives as extra load precisely when the service is already failing, and that is the feedback the next lesson closes.
The material
Where the attempt count comes from, what it buys, and the two shapes of duplicate work the same series also counts.
A retry policy of r retries makes up to r + 1 attempts at a request, stopping at the first success. If each attempt fails independently with probability p, the expected number of attempts per request is (1 − pʳ⁺¹)/(1 − p) and the probability the request eventually succeeds is 1 − pʳ⁺¹. The ratio of attempts to requests is the amplification factor.
The independence of attempts is a real assumption and it is generous. A request that failed because its payload is malformed will fail again with probability 1, and one that failed because a shard is down will fail again until the shard returns. Retrying a deterministic failure buys nothing and costs the full amplification, which is why policies distinguish retriable errors from the rest.
Eleven per cent more load for a hundredfold reduction in loss is an easy trade, and it is the trade every default retry configuration is quietly making. The difficulty is that the eleven per cent is not a constant.
Read the two columns together. At a 90% failure rate the policy has nearly tripled the load and is delivering just over a quarter of the requests; at 99% it has tripled the load to deliver three per cent. The load multiplier saturates at r + 1 while the benefit collapses, so the policy is at its most expensive and least useful in the same regime.
The same series counts something else. If an attempt can succeed at the server and still be retried — a timeout on the response rather than on the request — then every attempt beyond the first is duplicate work at the server, and the expected duplicates per request are attempts − 1 = p + p² + … + pʳ. At p = 0.1, r = 2 that is 0.11, or 9.91% of the server's work. This is one line of the geometric series rather than a separate model, and it is the reason at-least-once delivery needs idempotent handlers.
The factor compounds with depth. A request crossing three tiers, each retrying twice at p = 0.1, produces 1.11 attempts at the first tier, 1.11² at the second and 1.11³ ≈ 1.368 at the third: a 37% surcharge on the innermost service from a policy that looked like 11% at each hop. When the innermost service degrades, every tier above it begins retrying at once and the product is what arrives.
The standard mitigation is a cap on the fraction of a client's traffic that may be retries — typically a few per cent — enforced across the client rather than per call site. At p = 0.1 and r = 2 the retry share is 9.91%, already over such a budget, and the budget is what stops the share reaching 66.33%, which is where p = 0.99 puts it. The cap does not make retries cheaper; it makes the amplification bounded by something other than r + 1.
The last figure is the one to carry into the next lesson. A policy whose cost rises with the failure rate, in a system whose failure rate rises with load, is a loop. Nothing in this lesson closes it; the amplification is computed here against a failure probability handed in from outside.
Interactive
Expected attempts, success, and the load the policy adds
Method
Two numbers come out of the same series, and the fourth step is the one that stops the policy being sized against the wrong day.
An attempt must have an independent chance of succeeding. A malformed request, a rejected credential or a permanently missing key fails identically every time, so the policy pays the full amplification for a success probability of zero. Retry timeouts and transient errors; do not retry a deterministic failure.
attempts = (1 − pʳ⁺¹)/(1 − p) and success = 1 − pʳ⁺¹. Write both down together: quoting the success without the attempts is how a policy gets adopted without its cost being noticed.
λ × attempts is what the service actually receives. At 500 rps and 1.11 attempts that is 555 attempts a second, of which 9.91% is retry traffic — and along a three-tier chain the factors multiply rather than add.
Recompute at p = 0.5 and p = 0.9. The healthy-day figure decides nothing; the bad-day figure decides whether the policy is survivable. If λ × (r + 1) exceeds capacity, the policy has a worst case the system cannot serve.
A team proposes three retries rather than two, on the grounds that the extra attempt is almost never used. That is true on the healthy day, and the healthy day is not what the capacity has to cover.
The fourth attempt costs 0.000125 of an attempt on the healthy day and 0.729 on the bad one, which is the whole argument. Choosing r on healthy-day evidence sizes the policy against the case where it does nothing; the number that has to fit inside the capacity is λ × (r + 1), and at r = 3 that is four times the offered load.
Note what the bad-day success column says about the benefit as well. Going from r = 2 to r = 3 at p = 0.9 raises the success rate from 27.1% to 34.39% while raising the load ceiling from 6000 to 8000. Both directions are worse than they look on the healthy day.
For a faded rehearsal, the service has a capacity of 5000 rps and an offered load of 2000. The supplied first move is that the binding constraint is the ceiling λ × (r + 1), not the expected attempts. Find the largest r whose worst case fits inside capacity, then compute the success probability that choice gives at p = 0.5 and at p = 0.9, and check both against the lab.
You should be able to compute attempts and success from the same geometric series, convert an amplification into a load, evaluate the policy at a bad-day failure rate, and say why the factors multiply rather than add along a chain of tiers.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.