Definition · k-of-n availabilityA system is k-of-n when it serves as long as at least k of its n components are up. If the components are independent and each is up with probability A, then A(k, n) = Σ_{j=k}^{n} C(n,j) Aʲ (1 − A)ⁿ⁻ʲ. The term at j is the probability that exactly j are up: C(n,j) ways to choose which, Aʲ for those being up and (1 − A)ⁿ⁻ʲ for the rest being down.
n = 5, A = 99/100, listed term by term
j C(5,j) term value
0 1 (1/100)⁵ 0.0000000001
1 5 5(99/100)(1/100)⁴ 0.0000000495
2 10 10(99/100)²(1/100)³ 0.0000098010
3 10 10(99/100)³(1/100)² 0.0009702990
4 5 5(99/100)⁴(1/100) 0.0480298005
5 1 (99/100)⁵ 0.9509900499
───────────
1.0000000000
3-of-5 = j ≥ 3 = 0.999990149 = 99.9990149% five nines
The column sums to exactly 1, which is the cheapest available check on a binomial: a set of terms that does not sum to one has a wrong coefficient somewhere in it. The lab prints that sum for whatever n you set, and the coefficients are exact integers, so the twelfth row of Pascal's triangle is right rather than nearly right.
The two ends are the two previous lessons
At k = n only the j = n term survives, so A(n, n) = Aⁿ, which is the series product of n identical components. At k = 1 every term except j = 0 survives, so A(1, n) = 1 − (1 − A)ⁿ, which is the parallel formula. Three formulas, one of them; the lab computes all three and prints their agreement.
Example · Five replicas, three quorumsEach replica 99%. 1-of-5 is 99.999999990000% — ten nines, and the parallel formula to the digit. 3-of-5 is 99.9990149% — five nines, 5 min 10.65 s a year. 5-of-5 is 95.0990% — one nine, 17 d 21 h a year, and identical to (99/100)⁵. The same five machines span nine orders of magnitude in downtime depending only on how many of them have to agree.
When another replica lowers availability
Compare 5-of-5 at 95.0990% with a single machine at 99%. Four replicas were added and the availability fell by a factor of about fifty in downtime, because each new replica was also a new thing that had to be up. This is the honest answer to “should we add another node”: it depends on whether the quorum rule holds k fixed as n grows.
Majority quorums, k = ⌈(n+1)/2⌉, sit between the two extremes and do improve with n, because k grows at half the rate n does: 2-of-3 is 99.97020% and 3-of-5 is 99.9990149% at the same per-node 99%. That is the arithmetic behind five-node clusters, and it is also why going from five to seven buys much less than going from three to five.
Example · The write quorum that made things worseA team runs three replicas and, worried about durability, changes writes to require acknowledgement from all three rather than two. Write availability moves from 2-of-3, 99.97020%, to 3-of-3, 97.0299% — from three nines to one, and from 2 h 36 min to 10 d 20 h a year. Nothing about the hardware changed. The lab reproduces this in two slider moves: set n = 3, then k = 2 and k = 3.
One boundary to keep in view: this lesson counts replicas that are up and says nothing about whether they agree on the data. A quorum that is available can still return a stale read, and the rule that prevents that — read and write sets overlapping — is a different condition on k that belongs to the replication course.