1
Put a ± on a measured ratio
The standard error of an indicator from its window size, the n needed to resolve a tenth of a per cent, and the reason a short window’s 99.95% is not evidence of meeting a 99.9% objective.
Course · System Design
Measuring Systems
Every earlier course produced a number; this one asks how you would know it from a live system, and shows the four ways the measurement lies: percentiles that were averaged, histograms with wide buckets, load generators that skipped the slow requests, and scrape intervals that never saw the spike. It ends with the arithmetic of sampling, cardinality, load-test length and alerts.
SE = √(p(1−p)/n) 1 000 requests cannot tell 99.9% from 99.8%
fleet p99 = p99 of the merged sample, never the mean of per-host p99s
P(catch a spike of length d) = min(1, d/I)
P(see the top 0.1% in n) = 1 − 0.999ⁿ 95% needs about 3 000
pages/week = P(false alarm per window) × windows
What this course is about
- Subject
- System Design
- Lessons
- 10
- Format
- Interactive
Overview
What you will be able to do
By the end you can put an error bar on an indicator of service quality, aggregate percentiles without inventing one, correct two measurement biases, and size a sample, a load test and an alert so that each sees what it claims to see.
2
Aggregate and bucket percentiles honestly
The fleet p99 from merged samples against the mean of per-host p99s on the same data, and the interval a histogram’s p99 is actually known to.
3
Correct the two biases a measurement introduces
Coordinated omission, corrected by imputing the requests a stalled generator never sent; and the spike a scrape interval cannot see, with the longest invisible one named.
4
Size a sample, a test and an alert
The sampling rate that captures a rare event, the number of requests a load test needs to see its p99.9, the label cardinality a metric costs, and the false-alarm rate a threshold produces on a noisy ratio.
Lessons
10 lessons
The error bar on a ratio first, because every later lesson is about a ratio; then the three ways an aggregate lies; then the arithmetic of what a sample can and cannot see; and last the two alert calculations.
SLIs as RatiosCompute the standard error of a measured ratio, and the window size it takes to resolve a tenth of a per cent.Not marked
Percentiles Do Not AverageCompute a fleet p99 from the merged sample, and the mean of the per-host p99s on the same data, and say what the second one is.Not marked
Histograms and Bucket ErrorReport a percentile read from a histogram as the interval it is actually known to, and compute that interval's relative width.Not marked
Coordinated OmissionCompute the naive and the corrected percentile from one trace with a stall, and say which arrival model each one assumes.Not marked
Scrape Intervals and What They Cannot SeeCompute the probability a spike of a given duration is caught at a given scrape interval, and the longest spike that can hide entirely.Not marked
Sampling and Rare EventsCompute the chance a sampling rate catches at least one event of a rare problem, and the rate a stated confidence requires.Not marked
Metric CardinalityCompute the series count a set of labels produces and the bytes a day it costs, and price one more label before adding it.Not marked
How Long to Run a Load TestCompute the number of requests a run needs before a stated percentile has probably occurred in it at all.Not marked
Burn-rate AlertsCompute the fraction of an error budget a window consumes at a given burn rate, and the time to exhaustion, and choose the pairs that page at the right moment.Not marked
Threshold Alerts and False AlarmsCompute the exact probability that a fixed threshold fires on a healthy service, and the pages a week that produces.Not marked
Recommended background
Latency and the tail, and availability and failure, plus the binomial distribution, the geometric distribution and waiting times, the sum and product rules, variance and standard deviation and expected value from discrete mathematics.
Practice suggestions
Compute the standard error before arguing about any measured ratio. Most disagreements about whether an objective was met are disagreements about a number whose error bar contains both positions.
Do the aggregation lesson on data where the answer is surprising. The mean of per-host p99s can land below the fleet median, and the lab has a preset where it does.
Treat every alert threshold as a statement about the sample, not about the system. The last page computes what a threshold fires at when nothing at all has changed.
What it does not cover
Tracing systems, dashboard design and the operational tooling around metrics.
Statistical inference beyond these direct calculations — no hypothesis tests, no confidence intervals by a normal approximation, no regression.
Anomaly detection as a technique. “Threshold Alerts and False Alarms” computes a false-alarm rate; it does not propose a detector.