A threshold on a measured ratio is a statement about a sample. With 200 requests a window at a true error rate of 1%, a 2% threshold fires at five errors, and the exact binomial tail says that happens in 5.175% of windows — 104.32 pages a week, all of them false, from a service in which nothing at all has changed. Widening the window to 600 requests at the same threshold takes it to 5.71.
Core ideas
The threshold is compared against a measurement, the measurement is a sample, and samples fluctuate. Everything else follows.
“More than 2% errors” is evaluated against the ratio observed in one window, and that ratio is a random quantity even when the underlying rate is constant. With 200 requests at a true 1%, the observed count is Binomial(200, 0.01) with a mean of 2, and it exceeds 4 in one window in twenty. The alert is firing correctly; it is answering a question about arithmetic.
5.175% sounds like a tolerable rate until it is multiplied by the number of windows. Five-minute windows give 7 × 24 × 60 ÷ 5 = 2 016 of them a week, so the expected false pages are 0.05175 × 2 016 = 104.32. Fifteen a day. The per-window figure and the weekly figure are the same fact and only one of them is ever quoted.
Both work arithmetically and they differ in what they cost. Widening the window to 600 requests keeps the 2% threshold and drops the false rate to 0.849%, or 5.71 pages a week, because the ratio of a larger sample varies less — the 1/√n of the first page of this course, arriving as an alerting decision. Raising the threshold to 4% on the same 200 requests gives 0.021% and 0.43 pages a week, and buys it by refusing to detect a genuine doubling of the error rate.
The material
Where the firing count comes from, how a per-window probability becomes a weekly number, and the two levers with their prices.
A threshold alert fires when a statistic measured over a window crosses a fixed value. For an error ratio with threshold τ over n requests, it fires at the smallest integer error count strictly above τ·n. Its false-alarm probability is the chance of reaching that count when the true rate is the healthy one, and it is the upper tail of a binomial distribution.
Nothing has changed about the service. The true rate is 1%, the threshold is twice that, and one window in twenty crosses it anyway because five errors out of two hundred is an ordinary thing for a process with a mean of two to produce. This is not a tuning problem with a better threshold hiding behind it; it is what a threshold on a small sample does.
The second block is 1 − (1 − p)ᵏ for the third time on this course — the capture probability of a sampling rate, the chance a load test meets its tail, and now the chance of at least one false page in the next hour. Three lessons, one function, and the lab calls the same one in all three places so the claim can be checked rather than taken on trust.
A 47% chance of being paged in any given hour by a service in which nothing is wrong is the number that matters operationally. It is not merely wasteful: after a week of it the page has no information content left, and the response to the hundred and fifth one is indistinguishable from the response to a real outage, because there is nothing to distinguish it by.
200 requests, 2%. Fires at 5 errors, 5.175% of windows, 104.32 pages a week. This is the configuration almost everyone starts with.
600 requests, 2%. A fifteen-minute window instead of five, so the threshold now means 13 errors. 0.849% of windows and 672 windows a week: 5.71 pages. The threshold did not move; the sample got bigger and its ratio got less variable.
200 requests, 4%. Fires at 9 errors, 0.021% of windows, 0.43 pages a week. Quieter still — and it no longer fires on a true rate of 2%, which is double the healthy rate and is exactly the kind of change the alert existed to catch.
The two levers are not equivalent. Widening the window buys quiet with detection latency: the same regression is caught, ten minutes later. Raising the threshold buys quiet with detection sensitivity: some regressions are now never caught at all, at any latency. Latency is usually the cheaper currency, which is why the first move on a noisy alert should be the window.
There is a floor. A low-traffic service has few requests per window however long the window is, and at twenty requests a window no threshold on a ratio is stable: one error is 5% and two is 10%. For those services the honest answer is to alert on something else — an absolute count of errors, a budget consumed over a long window, or a synthetic probe — rather than to choose a threshold and live with the noise.
It is a false-alarm rate for a stated rule on a stated healthy rate. It is not a detector design: it says nothing about how quickly a real change is caught, and it does not compare this rule against any other. Those questions need the distribution under the alternative as well, which is the beginning of hypothesis testing and is outside this course.
What it does give you is the arithmetic to reject a rule before shipping it. If a proposed alert would fire a hundred times a week on a healthy service, that fact is available from four numbers — the true rate, the requests per window, the threshold and the window length — and it is available before anybody is woken up. The previous page’s burn-rate rule is the better-behaved alternative for exactly this reason: it thresholds a quantity that accumulates rather than a ratio that fluctuates.
Interactive
The exact chance a healthy service trips a fixed threshold, and the pages it costs
Method
Four numbers in, two numbers out, and the second one is the one to put in the review.
Not the threshold — the rate the service actually runs at, and how many requests fall in one evaluation window at its typical traffic. If traffic varies by time of day, do this twice: the quiet hours are where the false alarms live.
The smallest integer strictly above τ·n. At τ = 2% and n = 200 that is 5, not 4 — the off-by-one matters, because the tail at 4 is several times the tail at 5.
P(X ≥ k) for X ~ Binomial(n, p). This is the probability of a page per window when nothing is wrong, and it is the number people expect to be small and are surprised by.
7 × 24 × 60 ÷ w windows a week. A hundred pages a week is not a tuning detail; it is a rule that should not ship. Then try the window lever first and the threshold lever second, and say what each one gave up.
The service is healthy throughout: a true error rate of 1%, unchanging. The alert is an ordinary one, and every figure below is an exact binomial calculation on the numbers a team would actually have chosen.
The comparison line in the middle is worth keeping. Moving the firing count by one, from 5 to 4, changes the false-alarm rate by a factor of nearly three; from 5 to 6, by a factor of three the other way. A threshold on a small count is extremely sensitive to exactly where the boundary falls, which is another way of saying that a round-numbered threshold like 2% has no particular claim to being right.
The two fixes are not interchangeable even though both make the pager quiet. The wider window still detects a doubling of the error rate, ten minutes later than before. The higher threshold does not detect a doubling at all: at a true rate of 2% the mean is 4 errors in 200 and the alert wants 9. Quiet is easy; quiet while still detecting the thing you built the alert for is the actual problem.
For faded practice, take a service with a true rate of 0.5% and 100-request windows of one minute, with a 2% threshold. The supplied first move is that the mean is 0.5 errors and the alert fires at 3. Compute the tail, the windows a week and the pages, then say which lever you would pull — and whether a ratio threshold is the right instrument for a service this small at all.
You should be able to find the count a threshold fires at, compute the exact binomial upper tail at it, convert that into pages a week and into the chance of a page in the next hour, and say which of the window and the threshold you would change and what each one gives up.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.