Losing data needs all N copies gone inside one repair window, so to first order the annual probability per placement group is N! × fᴺ × (R/8760)ᴺ⁻¹: at three copies, a two per cent annual failure rate and a 24-hour window, 3.603 × 10⁻¹⁰. The repair window appears N − 1 times, so halving it divides the risk by 2ᴺ⁻¹ rather than by two. This is the one figure on the course that is rounded, and what is approximate is the model rather than the arithmetic.
Core ideas
Durability is a race between failures and repairs, and the shape of the answer comes from which of those two quantities appears how many times.
One copy failing is routine and costs nothing: the system notices and rebuilds it. Data is lost only when the remaining N − 1 copies also fail before the rebuild finishes. To first order the first copy fails at rate N f a year and each further one must follow inside R, giving N! × fᴺ × (R/8760)ᴺ⁻¹ as the annual probability per group.
f appears N times and R appears N − 1 times, so at three copies the risk is quadratic in the repair window: halving R divides the probability by four, not by two, and at four copies by eight. The rebuild rate is therefore a lever of the same order as the failure rate, and it is usually the one an operator can actually move.
At f = 2% and R = 24 h, two copies give 2.192 × 10⁻⁶ a year and three give 3.603 × 10⁻¹⁰ — a factor of about six thousand, not a factor of one and a half. Each copy multiplies the probability by f and by R/8760, both of which are small, so the improvement is exponential in N and the intuition of proportionality is wrong by orders of magnitude.
The material
Where the formula comes from, what it drops, why the dropped part is small here, and what the resulting number is and is not a statement about.
Consider a placement group holding N copies of an item, each on a device that fails independently at rate f a year, with a lost copy rebuilt within a repair window of R hours. The annual probability of losing the item is approximately N! × fᴺ × (R/8760)ᴺ⁻¹. This is a first-order rare-event estimate: it drops every term in which failures overlap or a repair completes mid-window.
The shape is easier to see than the constant. Any one of the N copies can fail first, at rate f each; the second must fail inside the window, which happens with probability about f × R/8760; the third likewise. Multiplying and counting the orderings gives the factorial, the N-th power of f and the (N − 1)-th power of the window fraction.
The exact fraction is printed beside the float deliberately. The arithmetic on rational f and R is exact and the two agree, which is how you can tell that the gap between this number and the truth is in the modelling rather than in the division. Every other figure on this course is exact; this one is rounded, and what rounds is the first-order truncation.
The estimate stands in for an exponential: the probability that a further copy fails within the window is 1 − e⁻ˣ with x = (N − 1) f R/8760, and the formula uses x itself. Here x = 1.095890 × 10⁻⁴ and 1 − e⁻ˣ = 1.095830 × 10⁻⁴, a relative gap of 0.0055%. The truncation is negligible in this regime and would not be if f or R were large, which is the condition under which the formula may be used at all.
Cutting R from 24 hours to 12 takes the annual loss probability from 3.603 × 10⁻¹⁰ to 9.007 × 10⁻¹¹, a factor of 2ᴺ⁻¹ = 4. At four copies the same halving is a factor of eight. Rebuild speed is not a second-order concern that gets attention after the replica count is settled; at three copies it is exactly as powerful as reducing the device failure rate by the same factor, twice over.
It is a per-group, per-year probability under a model in which devices fail independently at a constant rate and rebuilds always finish inside R. Multiply it by the number of groups to get an expected count for the fleet — 3.603 × 10⁻⁶ a year across ten thousand groups — and invert that for a mean time between losses. Those conversions are arithmetic; the model underneath them is the part that can be wrong.
And it usually is wrong in one specific direction. Independence is the assumption of “Correlated Failure”, and it fails here for the reasons it fails everywhere: three copies in one rack, on one power feed, in one firmware version, or written by one buggy release. The eleven-figure durability numbers quoted for storage systems are answers about device failure, and device failure is rarely what loses data.
A three-copy system at 3.603 × 10⁻¹⁰ per group per year would lose an item roughly once every 2.776 × 10⁵ years across ten thousand groups. Operators do not observe that; they observe losses from deletion bugs, from a bad deploy that corrupts all three copies identically, from an operator command applied fleet-wide, and from a rebuild that silently did not happen. None of those are in the model, and all of them are correlated — which is why backups with a different failure domain exist alongside replication rather than instead of it.
One consequence worth naming: the model says a rebuild that does not finish is not a slow repair, it is an infinite R. A group whose rebuild is stalled has left the regime the formula describes entirely, and the durability figure computed for the fleet does not apply to it. Monitoring under-replicated groups is therefore not an operational nicety; it is the condition under which the arithmetic is true.
Interactive
N failures inside one repair window, and what halving the window is worth
Method
The arithmetic is four multiplications. The first and the last steps are what decide whether the answer means anything.
The formula needs f R/8760 to be small — here 5.5 × 10⁻⁵. If the failure rate is high or the repair window is long, the truncated terms stop being negligible and the estimate overstates the risk, mildly, in a way that is no longer worth the simplicity.
f per year, R in hours, 8760 hours in the year. Express the window as the fraction R/8760 before raising it to a power; a window in minutes raised to the power N − 1 is wrong by a factor of 3600 at three copies.
N! × fᴺ × (R/8760)ᴺ⁻¹. Write the two exponents down: f appears N times and R appears N − 1 times. That asymmetry is the result, and it is what makes rebuild speed comparable in power to device quality.
Say that the figure is first-order, say what the truncation is worth, and say that it assumes independent device failures. A durability figure quoted to eleven significant figures with no assumption attached is a number about a fleet that does not exist.
A team is choosing between two and three copies, and separately has a rebuild window of seven days because the devices are large and the network is not. Both decisions are in the same formula and only one of them is being discussed.
Both levers are in view now and they are not the same size. The third copy buys a factor of about six thousand; the rebuild window, from a week to a day, buys forty-nine. Neither is negligible and the second is usually far cheaper, because it is a network and scheduling problem rather than a third of the storage bill.
The exponent is doing all of the work in the second comparison: 168/24 is seven, and seven squared is forty-nine because R appears N − 1 = 2 times. At four copies the same change would cost a factor of 7³ = 343, which is why systems that run wide erasure codes care about rebuild bandwidth to a degree that looks obsessive from outside.
For a faded rehearsal, keep three copies and a 24-hour window and suppose the device failure rate is revised upward from 2% to 5%. The supplied first move is that f appears N = 3 times, so the change is a factor of (5/2)³. Compute the new per-group probability and the fleet expectation, then find the repair window that would restore the original figure — and check both in the lab.
You should be able to evaluate N! fᴺ (R/8760)ᴺ⁻¹, say why halving R divides the risk by 2ᴺ⁻¹, state what the first-order truncation drops and roughly what it is worth, and name the correlated loss modes that sit entirely outside the model.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.