An interval is only worth what it guarantees. Chebyshev's inequality gives ±k standard errors a coverage of at least 1 − 1/k² for every distribution with a finite variance, with no shape assumed and nothing to be wrong about. At k = 5 that is 24/25 = 96%; the folklore ±2 SE guarantees 3/4 and not the 95% it is usually quoted with, because the 95% comes from an approximation this path has not built.
Core ideas
Three ideas, and the first is a distinction that most reported intervals blur: what an interval guarantees is not what it usually achieves, and it is the guarantee that can be relied on.
Chebyshev says at most 1/k² of the mass sits k standard deviations out, whatever the distribution is. Almost every distribution does far better than that, so the coverage you observe will usually beat the guarantee by a long way. Reading the bound as an estimate of the error is the standard mistake, and it runs in the safe direction exactly once: never.
It comes from the normal approximation, which this path has not built and does not assume. Under Chebyshev alone k = 2 guarantees 1 − 1/4 = 3/4. That is not a quibble about rigour: the approximation is a statement about sums of many comparable independent terms, and the estimators later on this course are sums of a few highly dependent ones.
Five standard errors against two is a width ratio of exactly 5/2, not of five. Buying that width back by running more instead — keeping k = 5 and narrowing SE — costs (5/2)² = 25/4 times the draws, because the width falls like 1/√n and a ratio of widths therefore squares.
The material
The one inequality this path can justify, the coverage it promises, and the two prices — in width and in runs — of promising it.
For a random variable X with mean μ and finite standard deviation σ, and any k > 0, P(|X − μ| ≥ kσ) ≤ 1/k². Applied to the sample mean, whose standard deviation is the standard error, this reads: the interval X̄ ± k·SE covers μ with probability at least 1 − 1/k².
The statement is Discrete Probability's, in “Variance and Standard Deviation”, which states it and stops. The two-line derivation from Markov's inequality — apply Markov to (X − μ)² — belongs to the Algorithms path, in “Randomised Algorithms”, in the lesson that turns an expectation into a probability. This page uses the statement and points at the proof rather than repeating either.
What it buys is unusual and worth naming. The inequality holds for every distribution with a finite variance: skewed, bounded, discrete, bimodal, anything. There is no assumption in it that can turn out to be false about the model you actually simulated, which is precisely what makes it the right tool for a simulation whose output distribution nobody has written down.
|X̄ − μ| ≥ k·SE is the same statement as (X̄ − μ)² ≥ k²·σ²/n, and the second one has no root in it. So the panel counts coverage by comparing two exact fractions, replication by replication, and the count is a whole number over a whole number rather than a verdict about two decimals.
On the table 1, 4, 6, 4, 1 over 16, whose variance is exactly 1, each replication of 100 draws has a mean with variance 1/100. At k = 5 the guarantee is 1 − 1/25 = 24/25 = 96%, and the band actually covered 40 of 40 — 100%. The gap between 96% and 100% is not luck; it is the size of the slack in a bound that had to hold for every distribution at once.
Drag k down to 1 and the guarantee becomes 1 − 1/1 = 0: Chebyshev promises nothing whatever about ±1 SE. The observed coverage at that width on the same forty replications is 25 of 40, or 62.5%. Both numbers are correct and they are answers to different questions, and confusing them is the entire subject of this page.
The phrase to avoid is “five times wider”. Five standard errors against two is a ratio of 5/2, and the number 5 in k = 5 is not a width ratio at all. The two-and-a-half is the honest price of the guarantee, and it is small enough that a reader who wanted 96% rather than an assumption of 95% might well pay it.
The other way to pay is with runs. Keep k = 5 and narrow the standard error until the band is as tight as ±2 SE was: since the width falls like 1/√n, that takes (5/2)² = 25/4 times the draws. Six and a quarter times the compute for the same width, with a guarantee instead of an assumption, is a trade a reader can actually evaluate — which is more than can be said for an unlabelled 95%.
On this table the half-width comes out at 1/2 and the folklore half-width at 1/5: both rational, because σ² = 1 and 100 is a square. Switch the panel to the skewed table, whose variance is 367/256, and the half-width becomes (1/32)√367 ≈ 0.59866, printed as a surd and then as a rounded decimal. The guarantee 24/25, the width ratio 5/2 and the run cost 25/4 do not move, because none of them contains a root.
That is the shape of every honest report on this course: the quantities that carry the argument stay rational, and the one quantity that has to be a root is printed in exact form with its rounding named. A coverage count of 40 out of 40 is a count; 96% is 24/25; and 0.59866 is (1/32)√367 shown to five places and no more.
Interactive
±k SE has guaranteed coverage 1 − 1/k², with nothing assumed and a width that says so
Method
Five steps, and the fourth is the one that keeps the report honest.
1 − 1/k² inverts to k = 1/√(1 − coverage). Ninety-six per cent needs k = 5; ninety per cent needs about 3.163; seventy-five per cent needs k = 2. Choosing k after seeing where the replications fell is choosing the interval that flatters the run.
SE = √(s²/n), carried as a surd. The half-width is k·SE, which is a second quantity with the same root in it; both are printed in exact form with the decimal labelled.
24/25 and 96% are the same number, and the fraction is the one that shows where it came from. A reader who sees 1 − 1/5² can check it; a reader who sees 96% has to trust it.
“Chebyshev, so this holds for any finite-variance distribution” is a claim a reader can evaluate. “A 95% confidence interval” on a simulation whose output law nobody has characterised is a claim about a normal approximation that has not been justified.
The observed coverage is a measurement of this run; the guarantee is a property of the method. Printing both, labelled, is what stops the first from being quietly promoted into the second.
Forty replications of a hundred draws each, from the table with μ = 2 and σ² = 1. The variance of a replication mean is therefore exactly 1/100, and the whole comparison below is in fractions.
The k = 1 row is the one to sit with. The guarantee is exactly zero — Chebyshev promises nothing at one standard error — and the band still covered sixty-two and a half per cent of the replications. Both numbers are right. One is a property of the inequality and one is a fact about forty runs of one table, and an interval reported without saying which is which is an interval that cannot be checked.
The k = 2 row is the one that gets misreported in practice. Its guarantee is 3/4. The 95% usually attached to ±2 SE is a normal-approximation figure, and on a simulation output it is an assumption about a distribution nobody has examined. Where the approximation is justified it is a good deal; the error is not using it, it is using it silently.
For a faded rehearsal, drag the draws per replication from 100 to 225 with k at 5. The supplied first move is that the standard error becomes 1/15 — 225 is a square, so no root appears — and the half-width becomes 1/3. Say what happens to the guarantee, what happens to the observed coverage, and which of the two you could have predicted before pressing anything.
You should be able to state Chebyshev's inequality and say where its statement and its proof live, convert a required coverage into a k and a k into a coverage, compute a half-width in exact form with its rounding named, give both prices of a wider guarantee — the k/2 in width and the (k/2)² in runs — and keep an observed coverage and a guaranteed one apart in the same sentence.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.