A request that needs every one of its dependencies succeeds only if all of them do, so under independence the availabilities multiply: A = ∏Aᵢ. Ten dependencies at 99.9% each give 99.00449%, which is two nines rather than three and nearly ten times the downtime of any single link. The phrase “as available as the weakest link” describes the wrong shape entirely.
Core ideas
One multiplication, one assumption, and one conclusion that sounds wrong until the numbers are in front of you.
If a request needs the auth service and the database and the cache and the network between them, it succeeds only when every one of them is up. The probability of an AND of independent events is the product of their probabilities, so A = A₁A₂…Aₙ. Nothing in that argument is about software; it is the definition of independence applied to the list of things a request cannot do without.
Each Aᵢ is at most 1, so multiplying by it cannot raise the running total. The chain is therefore worse than its weakest link, not equal to it: ten links at 99.9% give 99.00449%, whose downtime is 9.96 times the 43 min 48 s a single link permits. The unavailabilities add, near enough, and that is the useful way to hold it.
A = ∏Aᵢ is exactly right when the components fail independently and merely plausible otherwise. Two services in the same rack, behind the same certificate or shipped by the same deploy are not independent, and the product will read optimistically for one shape of correlation and pessimistically for another. “Correlated Failure” measures the gap.
The material
What multiplies, what it multiplies to, and what happens when a very good service is added to a chain that is already mediocre.
Components are in series for a request when the request needs all of them. If they fail independently, the availability of the whole is the product of the parts: A = ∏ᵢ Aᵢ. The unavailability is 1 − ∏ᵢ Aᵢ, which for small unavailabilities is close to Σᵢ (1 − Aᵢ) — the outages add up.
The approximation is worth carrying separately from the formula, because it is how the number is estimated in a meeting. Ten dependencies each down one part in a thousand are down roughly ten parts in a thousand altogether, so the chain is about 99%. The exact answer is 99.00449%, and the estimate was reached without a calculator.
The product is exact. Ten fractions of 999/1000 multiply to 999¹⁰ over 1000¹⁰, a thirty-digit rational, and the lab carries it as one rather than as a double that has already lost the digits the lesson is about. The percentage appears once, at the end.
This is the consequence people resist. A chain at 99.00449% gains a new dependency at 99.99% — a service ten times better than any link already in it. The chain becomes 98.99459%, which is one nine. Multiplying by 0.9999 is still multiplying by something less than one, and there is no availability, short of exactly 1, that a chain can absorb for free.
(999/1000)¹⁰ × (9999/10000) = 98.99459%, 7 h 20 min 22.24 s a month against the 7 h 16 min 2.05 s before it, and the nines count drops from two to one. Type the eleventh dependency into the lab's chain and watch the running product step down rather than hold still. Every architecture review that adds “just one more service, and it's a very reliable one” is this calculation.
Two readings of the same fact are useful in different rooms. The first: a chain is a budget and each dependency spends part of it, so a three-nines promise cannot be built from three-nines parts. The second: the cheapest way to raise a chain is usually to shorten it, because removing a link multiplies the product by more than any plausible improvement to a link that stays.
For the whole chain to reach 999/1000, ten equal links must each satisfy A¹⁰ = 0.999, which is A ≈ 0.9998999 — about four nines each. Set the lab's chain to ten entries of 99.99 and the product is 99.90004%: ten four-nines services, assembled in series, are worth three nines. That is the arithmetic behind every complaint that the platform team's SLOs are stricter than the product's.
One caution before the lab. Everything above assumes each dependency is genuinely required. A service called on a cache miss, or only for one endpoint, is not in series with every request and does not belong in the product at its full weight; putting it there produces a pessimistic number that the next argument will rightly dismiss.
Interactive
Ten dependencies at 99.9% are 99.0%, which is worse than any one of them
Method
The multiplication is trivial. Deciding what belongs in the list, and saying what you assumed about it, is the work.
Walk one request end to end and write down every component whose failure fails it. Anything consulted optionally, behind a cache or on one endpoint only, is not in series at full weight and should be left out or weighted down.
Carry each as a fraction and take the product. Ten links at 999/1000 are 999¹⁰/1000¹⁰; convert to a percentage once, at the end, and keep enough digits that the nines are still visible.
Turn 1 − A into minutes over a stated period and put it beside the downtime of the worst single component. The ratio should be close to the number of links; if it is not, either a link is dominating or something is in the list that does not belong.
The product is exactly right under independence and optimistic under a shared cause. Name what the components share — rack, power feed, deploy pipeline, certificate authority — and record the product as an upper bound where they share something that matters.
A real chain is not ten identical links. The arithmetic is the same and the reading changes: one component turns out to be spending most of the budget, and that is the finding.
The last block is the one to keep. The unavailabilities sum to 0.0027 and the exact 1 − A is 0.0026975; the sum is a good estimate and a better explanation, because it says immediately that auth and payments are three quarters of the outage and that improving the edge cannot matter. Two nines, and the ratio of 2.70 against the weakest link says the chain behaves like about three equal links.
For a faded rehearsal, the team proposes adding a fraud-check service at 99.995% to this path. The supplied first move is that its unavailability, 0.00005, joins the sum above. Predict the new chain availability and its monthly downtime before computing them, then type the six values into the lab and check. Then answer the question the number raises: is the new chain still inside a two-nines promise, and what would have to be removed to keep it inside a three-nines one?
You should be able to multiply a chain exactly, estimate it by summing unavailabilities, say how far under the weakest link it lands and why, work out what per-component availability a target demands, and name the independence assumption unprompted.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.