1
Put a floor under a route and read a critical path
The light-speed minimum for a distance, the fraction of a measured round trip it explains, and the longest path through a stage graph — the only stages whose speed-up changes the answer.
Course · System Design
Latency and the Tail
A request is slow for one of four reasons — it went far, it went back and forth, it waited for the slowest of many, or it was the unlucky one in a hundred — and each has a number. The course puts a floor under latency with the speed of light, counts round trips, computes percentiles by rank, and builds the budget that a timeout and a retry have to fit inside.
RTT_min = 2d / (⅔c) NY–London ≈ 56 ms, whatever you buy
t = k·RTT + bytes/bandwidth k is usually what matters
p99 = the ⌈0.99n⌉-th sorted value a rank, not an average
P(all n ≤ p99) = 0.99ⁿ ½ at n = 69
p99(A + B) < p99(A) + p99(B)
What this course is about
- Subject
- System Design
- Lessons
- 11
- Format
- Interactive
Overview
What you will be able to do
By the end you can put a floor under a route, read a percentile off a sample, say what a fan-out does to it, add two stages correctly, and lay out a budget that a timeout and a retry fit inside.
2
Compute percentiles from a sample, exactly
p50, p95 and p99 by nearest rank, and the reason the mean is none of them and cannot be made into one.
3
Quantify what fan-out and hedging do to a tail
The probability that all n independent calls clear their p99, the n at which a p99 becomes the median, and the tail and extra load a hedge at the p95 buys.
4
Add stages and lay out a budget
The p99 of a two-stage sum by convolution — below the sum of the p99s — then a budget allocated backwards from an SLO with a timeout and retry count that fit inside it.
Lessons
11 lessons
The floors first (distance, round trips, the critical path), then the tail as an object, then what composition does to it, and last the budget that everything has to fit inside.
Latency Is Not ThroughputClassify a transfer as latency-bound or bandwidth-bound, and price the two improvements before choosing one.Not marked
The Speed-of-Light FloorCompute the light-speed minimum round trip for a route and the fraction of a measured RTT it accounts for.Not marked
Round Trips, Not BytesCompute time-to-first-byte from a round-trip count, and the payload size at which bytes overtake round trips.Not marked
Serial Sums, Parallel MaxesCompute end-to-end latency through a stage graph, name the critical path, and read the slack on every stage off it.Not marked
Percentiles from a SampleCompute p50, p95 and p99 of a sample exactly by rank, and say what the mean is and is not about.Not marked
Tail Amplification under Fan-outCompute the slow-request probability for a fan-out of n calls, and find the n at which a per-call p99 becomes the request’s median.Not marked
Hedged RequestsCompute the hedged p99 and the extra load for a chosen hedge delay, and say when the independence it rests on is absent.Not marked
Percentiles Do Not AddCompute the p99 of a two-stage sum by convolution and compare it with the sum of the two stage p99s.Not marked
Latency BudgetsAllocate a budget backwards from an SLO across a stage graph and report the residual per stage, including where it is negative.Not marked
Timeouts and Retries in the BudgetChoose a timeout and retry count that fit a budget, and report the fraction of good calls the timeout kills.Not marked
Packet Loss and ThroughputCompute the loss-limited bound on one TCP flow and the number of parallel streams needed to fill a link.Not marked
Recommended background
Capacity estimation, plus independence, computing probabilities, random variables, cartesian products and tuples, and graph traversal from discrete mathematics, and roots and radicals from algebra.
Practice suggestions
Sort the sample by hand once. The nearest-rank percentile is a position in a sorted list, and a reader who has done it once stops treating p99 as an average for good.
Read the fan-out and the convolution labs together. They are the same fact twice — independence composing — once making the tail worse and once making it better than the naive sum.
Keep milliseconds throughout. Mixing microseconds into a budget is the one arithmetic error this course reliably produces.
What it does not cover
Where the tail comes from. Queueing is Queues and Utilisation; this course measures the tail and composes it.
How the measurement could be lying. Coordinated omission, bucket error and scrape intervals are Measuring Systems.
Congestion control as a subject. One bound — Mathis — is computed in “Packet Loss and Throughput” because it decides cross-region throughput; the algorithms behind it are not taught.