Every formula on this course so far assumed independence. Add a shared cause that takes both machines at once with probability c and the pair is down with probability c + (1 − c)p², not p². A common cause of one part in a thousand makes a pair whose independent product is 10⁻⁴ eleven times worse, and at c = p/(1 + p) the second machine is buying nothing at all.
Core ideas
A lab cannot check an assumption. What it can do is compute the answer under two models and put them on the same axes, which is what this one does.
Split the ways a pair goes down: a shared event takes both, with probability c; otherwise, with probability 1 − c, they fail independently and both happen to be down, with probability p². So P(both down) = c + (1 − c)p². The first term enters at full size while the second is shrunk, which is why the curve leaves the independent line the moment c does.
For p = 1/100 the independent pair is 10⁻⁴. A common cause at 10⁻³ — one chance in a thousand of an event that takes both — is ten times larger than that, and it raises P(both down) to 0.0010999, eleven times what independence promised. The pair falls from four nines to two. Correlation does not need to be likely to dominate; it needs only to be likelier than a product of two small numbers.
Solving c + (1 − c)p² = p gives c = p/(1 + p), exactly. Above it the pair is less available than a single machine. At p = 1/100 that is 1/101, about one per cent — so if a single shared event has a one per cent chance of taking both machines, the whole argument for the second machine has gone.
The material
How to write down a correlation you cannot measure precisely, what it does to the numbers from the last three lessons, and how to report a figure honestly when the assumption under it is doubtful.
A common cause is an event that takes more than one component at once: a power feed, a rack, a network partition, an expired certificate, a configuration push, a deploy. Writing c for the probability that such an event occurs in the period and p for each component's own independent failure probability, a pair is down together with probability c + (1 − c)p². c is a probability in [0, 1] and never a count of machines.
At c = 0 this is exactly independence and the pair is p², which is the parallel formula of two lessons ago. The model does not replace the earlier one; it contains it, and the earlier one is the special case where a claim that was never checked happens to be true.
Read the middle row first. When c equals p² the correlated answer is exactly twice the independent one, which gives a usable rule of thumb: the independent model is trustworthy only while the shared causes are much rarer than the product of the individual failures. For a pair of four-nines machines that product is 10⁻⁸, and very little in a data centre is rarer than that.
A pair is exactly as available as one machine when c = p/(1 + p).
The pair is no better than a single machine when c + (1 − c)p² = p.
Rearranging: c − cp² = p − p², so c(1 − p²) = p(1 − p).
Factor 1 − p² = (1 − p)(1 + p) and divide by 1 − p, which is positive for p < 1: c(1 + p) = p, hence c = p/(1 + p).
At p = 1/100 this is 1/101 ≈ 0.009901. Above that value the second machine is worse than useless: it adds cost, complexity and a failover path while lowering availability.
□
You will rarely know c. What you can do is bracket it: compute the independent figure, compute the figure at a c you would not be surprised by, and report both with the gap between them named. A single number computed under an unstated assumption is the failure mode this lesson exists to prevent, and it is not fixed by choosing a more conservative single number.
“Two 99% machines give 99.99%” is the independent answer. “Two 99% machines give 99.99% if they fail independently, and 99.89% if one event in a thousand takes both — they are in the same rack, so the second figure is the one to plan against” is the same arithmetic and a different claim. The second sentence is what the lab is for: move c up from zero and the red curve leaves the dashed line immediately.
The practical follow-up is that c is the quantity to attack. Splitting replicas across racks, power feeds, availability zones, deploy waves and certificate authorities are all attempts to lower c, and each of them is worth more than another replica once c is above p². The lab makes that comparison directly: a replica multiplies p² by p, and it does not touch c at all.
The shared deploy is the one that catches teams with genuinely independent hardware: five replicas in five racks across three zones, all receiving the same bad configuration within ninety seconds. For that failure the fleet is one machine, c is the probability of shipping a bad change, and no amount of n in the previous lesson's binomial changes the answer. Staged rollouts are an attempt to make c smaller by making the deploy not simultaneous.
Interactive
A 0.1% common cause swamps a pair whose independent product is 10⁻⁴
Method
Three of these steps happen before any arithmetic. The assumption is the thing being checked, and it is not visible in the number.
Rack, power feed, network path, availability zone, deploy pipeline, configuration source, certificate authority, DNS zone, the same library version, the same on-call engineer. Anything on that list is a candidate common cause, and the deploy pipeline is the one most often forgotten because it does not appear on a topology diagram.
An order of magnitude is enough to decide the question. Ask how often a single event has taken the whole set out in the last two years, divide by the number of periods, and use that. A c you can defend to one significant figure beats an independence assumption you cannot defend at all.
p² for independence, c + (1 − c)p² for the pair. If c exceeds p² the correlated term dominates and the independent figure is not an estimate of anything. Report the pair of numbers, not the better one.
Another replica multiplies the independent term by p and leaves c untouched. Separating the failure domains lowers c itself. Once c is above p² the second of those is the only one that moves the answer, and the break-even c = p/(1 + p) says where the replica has stopped being worth anything at all.
The independent figure is on the design document. The operations record says the rack has lost power twice in three years, which is the number the design document does not contain.
The conclusion is not “the estimate was a bit optimistic”. The second replica, in this rack, makes the service less available than one replica would be, while doubling the cost and adding a failover path that can itself fail. Everything about the arrangement is justified by a product of two small numbers that is swamped by a term nobody wrote down.
Note what fixes it and what does not. A third replica in the same rack changes p³ and leaves c alone: still 0.0186, still worse than one machine. Moving the second replica to another rack attacks c directly, and that is the whole of the improvement.
For a faded rehearsal, keep p = 1/100 and suppose the second replica moves to a different rack on the same power feed, which you estimate cuts c to 1/500. The supplied first move is to compare that c with p² = 0.0001 before computing anything. Work out P(both down), the pair availability, and the factor by which it still exceeds the independent promise; then say what the next c to attack is.
You should be able to compute c + (1 − c)p², compare c with p² to decide whether the independent figure means anything, derive and use the break-even c = p/(1 + p), and report a correlated pair as two figures with the gap between them named.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.