1
Convert between nines, minutes and repair time
Downtime per month and per year from an availability, the availability from MTBF and MTTR, and the MTTR a target requires — which is usually the cheaper lever.
Course · System Design
Availability and Failure
Nines are a fraction, and everything else in this course is what happens to that fraction under composition: chains multiply it down, redundancy multiplies it up but only if the failures are independent, retries multiply load exactly when the system can least afford it, and three copies of a file are safe in proportion to how fast a lost copy is replaced. Every claim is a probability you compute.
downtime = (1 − A) × period 99.9% = 43.8 min/month
A = MTBF/(MTBF + MTTR) halving MTTR = doubling MTBF
A = ∏Aᵢ series A = 1 − ∏(1 − Aᵢ) parallel
P(both down) = c + (1 − c)p² a 0.1% common cause swamps a 10⁻⁴ pair
attempts = (1 − pʳ⁺¹)/(1 − p) amplification, worst when p is worst
What this course is about
- Subject
- System Design
- Lessons
- 13
- Format
- Interactive
Overview
What you will be able to do
By the end you can convert nines into minutes, compose availability in series and in parallel while naming the assumption that lets you, compute what a retry policy does to load, and put a number on both durability and blast radius.
2
Compose availability and say what you assumed
The product for a chain, 1 − ∏(1 − Aᵢ) for redundant paths, the binomial tail for k-of-n, and the common-cause term that makes all three optimistic when it is not zero.
3
Compute what retries do to load
Expected attempts and success for a policy, the fixed point of a retry loop that can sit above ρ = 1 when the un-retried load was fine, and the synchronised waves that backoff without jitter produces.
4
Put a number on durability and blast radius
The annual probability of losing every copy of a replicated item and what halving the repair window does to it; and the probability two tenants share every node under shuffle sharding.
Lessons
13 lessons
The fraction first, then composition in three shapes, then the two ways a system makes its own load worse, and last the two questions that are about sets of machines rather than one: durability and blast radius.
Nines and DowntimeConvert a target availability into minutes of downtime and back, over a period you have stated.Not marked
MTBF and MTTRCompute availability from the two lifetimes, and the repair time a target actually demands.Not marked
Series: Availability MultipliesCompute the availability of a chain of dependencies and show it falls below the weakest of them.Not marked
Parallel: RedundancyCompute a redundant set's availability, then subtract the time spent switching between paths.Not marked
k-of-nCompute the availability of a k-of-n quorum and say when another replica helps.Not marked
Correlated FailureCompute pair availability with a common-cause term and find the c at which redundancy stops paying.Not marked
Error BudgetsCompute a period's error budget in failed requests and the share one incident burns.Not marked
Retries and Request AmplificationCompute the attempts a retry policy costs per request and the success probability it buys.Not marked
Retry StormsIterate the load-failure-retry map to its fixed point and say whether that point is above capacity.Not marked
Backoff and JitterCount retries per second after an outage, from one seed, with jitter and without.Not marked
Load SheddingCompute goodput under a shedding threshold and compare it with goodput under collapse.Not marked
Replica Loss and DurabilityEstimate the annual probability of losing every copy of an item, and what halving the repair window is worth.Not marked
Blast Radius and Shuffle ShardingCount the overlap between two randomly drawn shards and separate degradation from outage.Not marked
Recommended background
Capacity Estimation and Queues and Utilisation. Discrete Mathematics: independence, the binomial distribution, combinations, conditional probability and expected value. Algebra: geometric sequences and series.
Practice suggestions
Say the independence assumption out loud every time you multiply. Four of this course’s formulas are exactly right under independence and badly wrong without it, and “Correlated Failure” is the lesson that measures how wrong.
Do the retry-storm iteration by hand for three steps. A fixed point is easier to believe when you have watched the first three iterates move away from where you expected.
Keep availability as a fraction, not a percentage, inside any calculation. The interesting differences on this course are in the fourth decimal place and percentages hide them.
What it does not cover
Incident process, on-call practice and chaos engineering as a discipline.
Consensus availability, which needs the quorum arithmetic of Replication and Consistency.
Failure-rate models over time. Every failure here has a constant rate; bathtub curves and wear-out are out of scope.