A write that waits for its followers is acknowledged when the last of them replies, so its distribution is the maximum of N − 1 reply times and its CDF is F(t) raised to the power N − 1. Maxima do not add: on a real reply distribution, three replicas cost 24 ms at the p99 against 16 ms for one, while “three replicas cost three times one replica” predicts 48 ms — exactly twice the truth. The asynchronous alternative charges no latency and risks un-replicated bytes instead.
Core ideas
One rule, one arithmetic fact about powers, and one honest account of what the alternative costs instead.
A synchronous write is acknowledged when every follower has acked, so its wait is the maximum of N − 1 reply times. Under independence, P(max ≤ t) = F(t)^(N − 1), because all N − 1 must land by t. The median follower is irrelevant; the write is held by whichever one is slow this time, and which one that is changes every request.
Raising a number below one to a power pushes it down, but nothing like as fast as multiplying a time by N pushes that up. Here F(16) = 0.992 for one replica and 0.992² = 0.984064 for two, which is still close to one — so the p99 moves from 16 ms to 24 ms, not to 48 ms. The penalty for replication is real and it is a fraction of the one intuition charges.
Replicating in the background removes the wait entirely and puts acknowledged-but-unreplicated data at risk instead: write rate × mean lag × bytes per write. At 2000 writes/s, a mean lag of 3.536 ms and 400 bytes a write that is about 2.83 kB living on the leader and nowhere else. The trade is latency against a durability window, and both sides of it are computable.
The material
Where F(t)^(N − 1) comes from, what it does to a p99, and what the asynchronous alternative puts at risk instead.
A synchronous write is acknowledged to the client only after every follower has applied it. If each follower's reply time is independent with CDF F, the write is acknowledged by t with probability F(t)^(N − 1), the CDF of the maximum of the N − 1 followers.
Independence is a real assumption and it is named here rather than buried. Two replicas sharing a rack switch, a hypervisor or a garbage collector do not fail to reply independently, and this lesson's arithmetic describes replicas that do.
Read the fourth row carefully, because it is the whole lesson. One replica is done by 16 ms in 99.2% of cases. Two replicas are both done by 16 ms in 98.4064% of cases — a little worse, not twice as bad — and that shortfall against 99% is what pushes the p99 out one step, to 24 ms.
The mean wait rises by about a third and the p99 rises by half, while the multiplicative claim charges three times. The reason is that latencies of parallel requests do not add — only latencies of sequential requests add. Three replicas contacted one after another really would cost something like 3×; three contacted at once cost the slowest of them.
Drop the wait and replicate in the background. The write's latency becomes the leader's alone, and the data that has been acknowledged to a client but reached no other machine is rate × lag × bytes: at 2000 writes/s, a mean lag of 3.536 ms and 400 bytes a write, 2000 × 0.003536 × 400 ≈ 2828 bytes, about 2.83 kB. If the leader's disk is lost in that instant, those writes are gone and their clients were told they succeeded. That is the durability window, it is small here, and it is a number rather than a feeling.
Neither option is free and both are computable, which is the useful position to argue a design from: 8 ms added to the p99 of every write, or 2.83 kB of acknowledged data living in one place. Which is worse is a question about the business; which is which is a question this page answers.
Interactive
The maximum of N − 1 replicas, enumerated exactly
Method
Four steps, and the third is the one that separates this from the arithmetic most people do in their heads.
A mean reply time cannot answer a question about a p99, and a maximum is almost entirely a question about the tail. What you need is the distribution, or at least several of its percentiles.
The leader's own apply is not a network wait and the write is not waiting for it in the same sense. Three replicas means two followers and an exponent of two; five replicas means an exponent of four.
F(t)^(N − 1) at each attainable t, then the smallest t where it reaches 0.99. Do not multiply a percentile by anything: the percentile of a maximum is not a function of the percentile of one replica, it is a function of the whole curve.
write rate × mean lag × bytes per write is the acknowledged data that exists in one place. State it in bytes and in seconds of writes, and say which failure would lose it.
A leader and two followers. The per-replica ack time is measured and written as value:weight pairs. The question is the p99 of a synchronous write, and the answer the team has already guessed is three times one replica's.
The 24 ms is not a small penalty and nobody should pretend it is: half as much again on the p99 of every write is a real cost, and it is the price of not having an unreplicated window. What it is not is 48 ms, and a team that budgeted 48 ms has given up a design option on the strength of arithmetic that was never right.
Notice which part of the distribution did the work. The move from 16 to 24 happened because 0.992² fell below 0.99, a shortfall of about eight thousandths. Everything below the 99th percentile — the 600 weight at 2 ms, the 300 at 4 ms — contributed nothing at all to the answer. Tail questions are answered by tails.
For a faded rehearsal, take N to 5 and then to 9 on the same distribution. The supplied first move is that the exponents are 4 and 8, so the quantities to test are 0.998⁴ and 0.998⁸. Predict both p99s before computing them, then check both in the lab, and say in one sentence why one of them did not move at all.
You should be able to raise a measured CDF to the power N − 1, read a p99 off the result, say how far the multiplicative claim is from it, compute the asynchronous alternative's window in bytes, and name the independence assumption the exponent rests on.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.