A read issued t after a write is stale exactly when the follower's lag exceeds t, so P(stale) = P(lag > t) — a tail, and nothing else. On a measured lag distribution a read at 50 ms is stale with probability 2/25, and a read at the mean lag of 31.45 ms is still stale one time in five. Read-your-writes and monotonic reads are the two guarantees that fail on precisely this mass.
Core ideas
Staleness is one subtraction away from a distribution you can measure. The difficulty is entirely in which number people reach for instead.
The follower has the write if its lag is at most t, and does not if the lag is more. So P(stale) = P(lag > t) — the sum of the probabilities above t and no other part of the distribution. On the measured lag here, a read 50 ms after the write lands in 2/25 of cases on a follower that has not caught up.
The mean lag on this distribution is 31.45 ms, and the mass strictly above 31.45 ms is 200 parts in 1000. Waiting for the mean therefore leaves one read in five stale, because a mean is a balance point and not a deadline. Any lag distribution with a tail behaves like this, and lag distributions are all tail.
Read-your-writes says a client that wrote sees its own write; it fails when that client's next read lands on a follower still in the tail. Monotonic reads says a client never sees time go backwards; it fails when a client is moved from a caught-up follower to a lagging one. Both are statements about this one number, which is why measuring it is what makes them arguable.
The material
What staleness is a probability of, why the mean cannot answer it, and what the tail costs at a real read rate.
Replication lag is the delay between a write committing on the leader and being visible on a follower. A read of that follower issued t after the write is stale if it returns the value from before the write, which happens exactly when the lag exceeds t. So P(stale at t) = P(lag > t).
Lag is not a constant and it is not well described by its average. It is a distribution with a long right tail, produced by whatever the follower is occasionally busy doing — a compaction, a slow disk, a network hiccup, a burst of writes it is still applying.
Every figure in that table is a sum of weights divided by 1000. There is no model being fitted and no parameter being estimated: if you can measure the lag, the staleness probability at any delay is a column of the same table.
“Average lag is fifty milliseconds, so reads after fifty milliseconds are fine” is the sentence this lesson exists to break, and the break is not subtle: at the mean of this distribution, twenty per cent of reads are wrong. Half the mass sits at 10 ms and pulls the average down while the 120, 250 and 400 millisecond events — the ones that actually produce the complaint — sit far above it.
A service serving 12 000 follower reads a second with a read-after-write delay of 50 ms produces 12 000 × 2/25 = 960 stale reads every second. That is the number to put in front of anyone who has described the problem as intermittent. It is not intermittent; it is eight per cent, all day, and the reason it is reported as intermittent is that most reads are not of something the reader just wrote.
The repairs are all versions of the same idea: make the read wait, or make it go somewhere that has the data. Route a client's reads to the leader for a while after it writes; pin a session to one follower so monotonic reads hold even though read-your-writes may not; or carry the write's version with the client and have the follower wait until it has reached that version. Each costs something, and this distribution is what says how much of it you need.
Interactive
P(stale) is the tail past the read delay, and the mean cannot see it
Method
The read delay is a design choice and the lag is a measurement. Keeping them apart is most of the discipline here.
A single lag number on a dashboard is usually a mean or a current value, and neither answers a staleness question. What you want is a histogram, or failing that a handful of percentiles including a p99 and a p999.
“Stale” is meaningless without a t. A page that re-reads immediately after a write has a t of a few milliseconds; one that re-reads after a redirect and a render has a t of a few hundred. These give wildly different answers from the same distribution.
That sum is P(stale), exactly. Resist substituting the mean, the median or a percentile for the delay: the question is about a specific t you chose, and the answer is the tail past it.
P(stale) is easy to dismiss and 960 stale reads a second is not. Report both, and say which guarantee — read-your-writes or monotonic reads — the failures will be reported as.
The lag of a read replica has been measured for a day and bucketed. The product question is how long the application must wait after a write before reading a follower is safe, and “safe” has to be given a number before it can be answered.
The middle column is the deliverable and the right-hand one is what gets the work scheduled. Nobody argues with 2/25; everybody argues with 960 complaints a second, and it is the same statement.
The mean row is there to be looked at. Waiting the average lag — which sounds like a generous, even wasteful, thing to do — leaves twenty per cent of reads stale, worse than waiting a flat 50 ms. That is what it means for a distribution to be right-skewed, and lag distributions always are.
For a faded rehearsal, suppose the team cannot wait 250 ms and proposes 120 ms instead. The supplied first move is that the tail past 120 is 60 + 15 + 5 = 80 parts in 1000. Wait — check that against the table before you trust it, state the correct P(stale) at 120 ms, convert it to stale reads a second at 12 000 reads/s, and say which of the two guarantees a user would report the failure as.
You should be able to compute P(lag > t) from a measured distribution at any delay, say why the mean cannot be used as a freshness delay, convert the probability into stale reads a second, and name which of read-your-writes and monotonic reads a given user report corresponds to.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.