Lesson · Measuring Systems · What a number means

SLIs as Ratios

An indicator of service quality is a ratio: good events over valid events, counted across a window. It is measured, not observed, so it arrives with an error bar whose width is √(p(1 − p)/n) and depends on the size of the window rather than on the health of the service. A thousand requests cannot tell 99.9% from 99.8%, and most arguments about whether an objective was met are arguments about a number whose error bar contains both positions.

Open the interactive lesson Practice

SLI = good events ÷ valid events, over a stated window SE = √(p(1−p)/n) n = 1 000, p = 0.9995 → SE ≈ 0.0707% gap to a 99.9% objective 0.05% = 0.71 SE inside the noise n ≥ p(1−p)/w² w = 0.025% → 7 996 requests precision goes as 1/√n 4× the requests buys 2× the precision

A ratio, and the error bar that arrives with it

Core ideas

What this lesson establishes

One formula, and the two things it is usually misread about: what is in the denominator, and what a difference smaller than the error bar is evidence of.

An SLI is a ratio, and its denominator is a decision

Good events over valid events. Both halves are choices you make before you measure anything: a request that a client abandoned, a health check, a request rejected for a malformed body — each is either valid or not, and moving one of them across the line moves the ratio without anything happening to the service. Two teams quoting different availabilities for the same week are usually quoting different denominators, and no amount of precision resolves that.

The ± is a property of the count, not of the service

Each valid event either succeeds or does not, so the count of good ones behaves like n independent trials at probability p, and the measured ratio has variance p(1 − p)/n. Its standard error is the square root, √(p(1 − p)/n). Nothing about the system appears in that expression except p itself: a healthy service measured over a short window produces a noisy number, and the noise is arithmetic rather than a symptom.

Precision costs the square

The error bar shrinks as 1/√n, so halving it costs four times the traffic and the rest of the day buys less than the morning did. At p = 0.999 one thousand requests give a standard error of about 0.1% — the entire distance between 99.9% and 99.8%, which is why a thousand requests cannot tell those two apart. Resolving a tenth of a per cent takes about four thousand, and resolving a hundredth takes a hundred times as many again.

The material

The ratio, its standard error, and the window that resolves a tenth of a per cent

Where √(p(1 − p)/n) comes from, what a gap of 0.71 standard errors is evidence of, and the request count a stated precision requires.

Definition · Service level indicator

A service level indicator is a ratio good ÷ valid of events counted over a stated window. The objective is a target the indicator is compared against. An indicator is not complete without its window and its denominator: “99.95%” names neither, and the same events can produce several different correct values of it.

Everything on this course is a ratio of this shape or a percentile of a sample, and both are measurements. The first thing to do with a measurement is to say how wide it is, because a ratio quoted to four figures from a five-minute window is three of those figures wider than it looks.

Theorem · The standard error of a measured ratio

Let each of n valid events be good with probability p, independently. The number of good events has variance n·p(1 − p), so the measured ratio good/n has variance

p(1 − p)/n, and standard error SE = √(p(1 − p)/n).

The variance of a Bernoulli trial, and the rule that the variance of a mean of n independent copies is the variance divided by n, are proved in “Variance and Standard Deviation” on the Discrete Mathematics path. This page uses the result; that page establishes it.

Independence is the assumption and it is the one that fails. Requests that fail because one dependency is down are not independent trials, and during an incident the true error bar is wider than this one — the formula is a floor on the uncertainty, not a description of it. What it is exactly right about is the quiet week, which is when the arguments happen.

a window of n = 1 000 requests reporting 99.95% p = 9995/10000 = 1999/2000 1 − p = 1/2000 p(1 − p) = 1999/4000000 p(1 − p)/n = 1999/4000000000 exact SE = √(1999/4000000000) = 0.000706929… ≈ 0.0707% rounded, and it is a root

The variance is an exact fraction and the standard error is not, because a square root of a rational is almost never rational. The lab prints the fraction under the root exactly and labels the root itself as rounded. That is the only rounded figure on this course and its lesson says so.

Now compare the measurement to the objective. The gap is 0.9995 − 0.9990 = 0.0005, which is 0.05%. In units of the error bar that is 0.0005 ÷ 0.000707 = 0.71 — less than one standard error. A window that reports 99.95% against a 99.9% objective has produced a number that a service sitting exactly on the objective would produce routinely.

Example · A window that says 99.95% and settles nothing

One thousand requests, 99.95% good, a 99.9% objective, and a meeting. The measured ratio is above the objective, so the objective was met — except that the same window run again on a service that is exactly at 99.9% would land above 99.95% about a quarter of the time by arithmetic alone. The honest report is “99.95% ± 0.07%, which does not separate us from the objective”. It is a shorter meeting.

How many requests resolve a tenth of a per cent

Turn the formula round. To resolve a difference of w, ask for an error bar of w: set √(p(1 − p)/n) ≤ w and solve for n. A difference worth acting on is usually wanted at two standard errors rather than one, so put half the gap in as w.

want SE ≤ w, so p(1 − p)/n ≤ w², so n ≥ p(1 − p)/w² gap of 0.05%, wanted at 2 SE: w = 0.00025 n ≥ (1999/4000000) ÷ (1/16000000) = 1999 × 4 = 7 996 requests check by search: the smallest n with 2·√(p(1−p)/n) ≤ 0.0005 → 7 996

Two routes to one integer, which is the check: the closed form and an exact scan over n agree on 7 996. At a hundred requests a second that is eighty seconds, which is a useful thing to know before anyone proposes a five-minute window as the unit an objective is judged in.

The 1/√n is not specific to measurement. Estimating a quantity by running a simulation many times has exactly the same error bar for exactly the same reason, which is what “Monte Carlo Estimation and the Standard Error” on the Operations Research path is about: four times the runs for twice the precision, whether the trials are simulated or served. No tail inequality is needed on this page — Markov and Chebyshev, which bound a deviation without knowing the distribution, are “From Expectation to Probability” on the Algorithms path, and here the distribution is known.

What the error bar does not cover

It covers sampling noise and nothing else. A denominator that silently excludes the requests that never reached the load balancer produces a ratio that is biased, not noisy, and doubling the window makes it more precisely wrong. The same is true of a window whose boundary falls in the middle of an incident, of retries counted as separate valid events, and of a client-side timeout the server never saw. Systematic error has no √n in it, and the only defence is writing down what the denominator is.

Interactive

An SLI is a ratio with an error bar

What a window of n requests can and cannot tell you about a ratio

Set the window and what it measured

The variance and the required n are exact fractions of the window and the ratio you set; the standard error is the square root of that fraction and is labelled as rounded. Push the window down to a few hundred requests and watch the error bar swallow the objective line, then find the n at which the two separate.

Requests in the window1000

Measured success ratio (ten-thousandths)9995

The objective

Measured SLI—

Variance p(1−p)/n, exact—

Standard error (rounded)—

Gap to the objective—

Gap, in standard errors—

Window needed to resolve it—

The variance is an exact fraction and so is the window size this page solves for: p(1−p)/n ≤ w² needs no square root at all. Only the error bar itself is rounded, because it is a root, and it is the one rounded figure on this course.

Method

Putting a ± on a ratio you were handed

The first two steps are arithmetic and the third is the one that changes what anybody says next.

Write down the denominator before the ratio

What counts as a valid event, and over what window. If you cannot say it in a sentence, the number is not yet a measurement of anything and the error bar is the smaller of your problems.

Compute p(1 − p)/n, then take the root

Keep the variance as a fraction and round only at the root. At p near one the product p(1 − p) is small and the arithmetic is easy: 0.999 × 0.001 = 0.000999, divided by the window size.

Express the gap to the objective in units of SE

Subtract, divide by the standard error, and say the answer out loud. Below about two, the window has not separated the measurement from the objective, and that is the finding — not a smaller finding than a clear pass or fail, a different one.

If the gap matters, solve for the window

n ≥ p(1 − p)/w² with w set to half the gap you want resolved. Then ask whether that many requests exist in the period you are allowed to wait, because often they do not and the objective has to be judged over a longer window instead.

Ratios, windows and error bars

Score 0 / 0

Choose the answer you can justify.

1 000 requests, 99.95% measured, a 99.9% objective

Every figure below except the root is an exact fraction, and the root is the only rounded number on the page. The question is not whether 99.95% is bigger than 99.9% — it plainly is — but whether this window can tell the difference.

p = 9995/10000 = 1999/2000 n = 1 000 p(1 − p) = (1999/2000)(1/2000) = 1999/4000000 variance = p(1 − p)/n = 1999/4000000000 SE = √(1999/4000000000) = 0.000706929… ≈ 0.0707% (rounded) gap = 0.9995 − 0.9990 = 0.0005 = 0.05% gap ÷ SE = 0.0005 ÷ 0.000706929 = 0.71 so the window reports 99.95% ± 0.07% and the objective 99.90% is 0.71 SE away — inside the bar for a 2 SE separation: w = 0.00025 n ≥ p(1 − p)/w² = (1999/4000000) × 16000000 = 7 996 exact scan over n agrees: 7 996

The last block is the one to keep. A window of 1 000 requests can distinguish 99.95% from about 99.81% and from nothing closer; to separate it from 99.9% takes eight times as many requests, and to separate 99.99% from 99.95% would take far more again.

For faded practice, take the same objective and a window of 5 000 requests reporting 99.92%. The supplied first move is p(1 − p) = (2498/2500)(2/2500); compute the standard error, express the gap of 0.02% in units of it, and say in one sentence what the window has established. Then set the lab to those values and check the error bar against the objective line before opening the quiz.

A last habit worth forming: when someone quotes an indicator to four significant figures, the first question is the window, not the figure. The digits past the error bar were produced by the division, not by the service.

Common mistakes

Reading a ratio over requests as a ratio over users“Our p99 is 200 ms, so 99% of users see under 200 ms” is the same slip in percentile clothing, and it is wrong in the direction that matters. A user who makes ten requests meets the slow tail with probability 1 − 0.99¹⁰ = 9.56%, not 1%, and a page that fans out to twenty backend calls is worse again. The indicator counts events; a user is a bundle of them, and “Tail Amplification under Fan-out” on the Latency and the Tail course is the arithmetic that converts one into the other.

Quoting more digits than the window supports99.9473% from a five-minute window is a division carried out to six figures on a measurement good to two. The digits are real arithmetic and they are not information, and once they are in a dashboard someone will compare two of them and find a difference. Round the report to the error bar and state the bar.

Changing the denominator between the two windows being comparedLast month excluded health checks and this month does not, so the indicator moved by more than any error bar explains and nothing happened to the service. Any comparison of two indicators is first a comparison of two denominators, and a change in definition should be recorded as an event on the graph rather than absorbed into it.

Check your understanding

✓

Finish when a difference smaller than the error bar reads as a finding, not as a result.

You should be able to state the denominator and window of any indicator you quote, compute √(p(1 − p)/n) from them, express the gap to an objective in units of it, solve n ≥ p(1 − p)/w² for a stated resolution, and say which of your error sources the square root does not cover.

Note: Every number after this one is a percentile rather than a ratio, and percentiles fail in a second way that has nothing to do with noise: they are routinely combined by an operation that is not defined on them. “Percentiles Do Not Average” takes three hosts whose individual p99s are 12, 14 and 400 ms and shows that the average of those three numbers lands below the fleet’s median.