Lesson · Measuring Systems · Sizing samples and alerts

Sampling and Rare Events

Sampling one request in a hundred does not show you one per cent of a problem. A problem made of ten requests is caught with probability 1 − (1 − s)¹⁰, which at one per cent is 9.56% — so it is missed nine times in ten, and reaching ninety-five per cent confidence takes a rate of 25.9%, not 10%. Sampling the tail instead fixes the capture problem and breaks the average, and the repair is to count each sampled request 1/s times.

Open the interactive lesson Practice

P(capture ≥ 1 of k at rate s) = 1 − (1−s)ᵏ s = 1%, k = 10 → 9.56% caught, 90.44% missed entirely 95% confidence on k = 10 needs s = 259/1000 — 26× the rate, not 10× keep everything over 100 ms and the mean reads 203.69 ms count each sampled request 1/s times and it returns to 9.34 ms exactly

What a sampling rate can and cannot see

Core ideas

What this lesson establishes

One formula and one repair. The formula is the same 1 − (1 − p)ᵏ that fan-out amplification uses, and it is about to be used twice more on this course.

A rate is a per-event probability, and a problem is many events

Sampling at s keeps each request independently with probability s, so a problem that manifests in k requests is missed entirely with probability (1 − s)ᵏ and seen at all with probability 1 − (1 − s)ᵏ. The intuition that a one per cent sample shows one per cent of everything is right about totals and wrong about existence: it gives s·k = 0.1 expected captures, and the expected number of captures being a tenth is exactly why the probability of any capture is small.

Confidence in a rare event costs far more rate than it looks

Solving 1 − (1 − s)¹⁰ ≥ 0.95 gives s ≥ 25.9%: to move from 9.56% confidence to 95% takes twenty-six times the sampling rate, not ten times. The relationship is exponential in k and nothing like linear in s, which is why “let us raise sampling from 1% to 10% and see” is a decision that buys 65% confidence and feels like it should buy more.

Sampling the tail is a good sample and a dishonest average

Keep every request over a threshold and sample the rest at s, and you capture every slow request while shrinking the volume — the right trade for finding problems. The sampled set is then not representative, so its plain mean is meaningless: on the lab’s traffic it reads 203.69 ms against a true 9.34. Dividing each kept request by its own inclusion probability before averaging recovers the true mean exactly.

The material

The capture probability, the rate it takes, and the mean sampling ruins

Why one per cent sampling misses a ten-request problem nine times in ten, what confidence costs, and how to average a sample that was deliberately biased.

Definition · Sampling rate, and capture

Under uniform sampling at rate s, each event is retained independently with probability s. A problem that produces k events is captured if at least one of them is retained. The capture probability is

1 − (1 − s)ᵏ,

which depends on k as strongly as it depends on s — a fact the phrase “one per cent sampling” hides completely, because it names only one of the two.

s = 1/100, k = 10 missed (1 − s)ᵏ = (99/100)¹⁰ = 0.904382075… 90.44% caught 1 − that = 0.095617925… 9.56% expected captures s·k = 0.1 the rule of thumb 1 − e^(−s·k) = 1 − e^(−0.1) = 9.516% differs from the exact 9.562% by 0.045 percentage points

The last two lines are a small lesson of their own. The 1 − e^(−sk) form is an approximation, and the lab computes it by a series that rounds at about 10⁻¹³; the gap between the two columns is 4.5 × 10⁻⁴, nine orders of magnitude larger. An approximation and a rounded approximation are different kinds of wrong, and only one of them gets smaller if you compute harder.

The rate a stated confidence needs

Turn it round: given k and a target confidence c, find the smallest s with 1 − (1 − s)ᵏ ≥ c. The lab scans the thousandths and reports the first that clears the bar, which avoids a logarithm and gives an exact fraction.

want 1 − (1 − s)¹⁰ ≥ 95/100 (1 − s)¹⁰ ≤ 5/100 scan s = 1/1000, 2/1000, … : s = 250/1000 (3/4)¹⁰ = 0.0563 → 94.37%, short s = 259/1000 = 0.0499… → 95.01%, first to clear so 25.9% sampling, against 1% — a factor of 26 for a factor of 10 in confidence

Ten per cent sampling, the obvious compromise, gives 1 − 0.9¹⁰ = 65.13% — better than a coin toss and not a guarantee of anything. If a problem must be visible whenever it occurs, uniform sampling is the wrong instrument at any rate you can afford, and the answer is to stop sampling uniformly.

Tail-biased sampling, and the average it breaks

Keep every request slower than a threshold and sample the rest. Now every slow request survives, which is what you wanted, and the retained set over-represents the tail by a factor of 1/s. Any statistic computed naively on the retained set is a statistic of a population that does not exist.

true traffic 5 ms × 900 8 ms × 60 12 ms × 30 400 ms × 10 n = 1 000 true mean (4 500 + 480 + 360 + 4 000)/1 000 = 9 340/1 000 = 9.34 ms keep all ≥ 100 ms, sample the rest at s = 1/100: retained 10 slow + 990/100 = 9.9 of the fast naive mean (4 000 + 53.4)/19.9 = 4 053.4/19.9 = 203.69 ms reweight: divide each retained request by its own keep probability slow counted ×1 fast counted ×100 (4 000 + 5 340)/(10 + 990) = 9 340/1 000 = 9.34 ms exactly

The reweighted figure is not an approximation of the true mean: it is the true mean, recovered exactly, because the weights are the reciprocals of known inclusion probabilities. This is the Horvitz–Thompson estimator, and it is the whole content of “remember to divide by the sampling rate”.

It does not rescue everything. A weighted mean is easy; a weighted percentile needs the weights carried into the rank, and a naive percentile of a tail-biased sample is far worse than a naive mean because it is the tail that was over-kept. If your pipeline samples non-uniformly, every statistic downstream of it needs to know the rate, and the rate has to travel with the data.

Example · Two teams, one trace pipeline

The platform team samples traces at 1% uniformly, so the trace store holds one request in a hundred and its averages are honest. A customer reports ten slow requests; the probability any of them is in the store is 9.56%, and nobody finds anything.

They switch to keeping every request over 100 ms plus 1% of the rest. Now all ten are there. A week later a dashboard built on the same store reports a mean latency of 203.69 ms and an incident is declared about a service whose true mean is 9.34 ms. Both problems are the same decision seen from two ends, and the fix is to store the rate alongside the request rather than to choose between the two failures.

Interactive

One per cent sampling does not see one per cent of a problem

The capture probability, the rate it takes, and the mean it ruins

Set the rate and the problem

The capture probability is exact, printed as a fraction beside the percentage, and the rate needed for a stated confidence is found by scanning the thousandths rather than by a logarithm. The table’s third column is the 1 − e^(−sk) rule of thumb, kept beside the exact answer so the gap between an approximation and a rounding is visible rather than asserted.

Sampling rate s (per mille)10

Requests in the problem you are hunting10

Confidence you want in catching one

The traffic (ms:count)

Sample every request slower than (ms)100

P(catch at least one)—

P(miss all of them)—

Rate needed for the target—

Mean of the tail-biased sample—

Reweighted estimate—

True mean of the traffic—

One per cent sampling does not see one per cent of every problem — it sees a problem of k requests with probability 1 − (1−s)ᵏ, which for small k is almost nothing. Keeping every slow request instead fixes that and breaks the mean, unless each sampled request is counted 1/s times when the mean is taken.

Method

Choosing a sampling rate for something you must not miss

The rate is a consequence of two numbers: how many events the problem makes, and how sure you need to be.

Say how many events the problem produces

k is the number of requests, traces or log lines the thing you are hunting will generate. A ten-request problem and a ten-thousand-request problem need completely different rates, and “rare” without a number cannot be sized.

Compute 1 − (1 − s)ᵏ at the rate you have

If the answer is under a half, your current pipeline is a coin toss you lose. Quote it as a probability of capture rather than as a sampling percentage, because the two numbers are wildly different and only one of them answers the question.

Solve for the rate your confidence needs

Scan s upward until 1 − (1 − s)ᵏ clears the target. Then price it: the rate multiplies storage and ingest directly, and 25.9% sampling is a quarter of full volume.

If the price is impossible, sample non-uniformly instead of lowering the bar

Keep everything that is already interesting — errors, slow requests, a named customer — and sample the rest. Capture becomes certain for the events you selected on, at a fraction of the volume.

Carry the rate with the data

Every record needs to know the probability it was kept with, or no downstream statistic can be corrected. Reweighting recovers a mean exactly; it can only do so if the weight survived the trip.

Rates, captures and weighted means

Score 0 / 0

Choose the answer you can justify.

One per cent sampling against a ten-request problem

The first half is the capture arithmetic and the second half is what the obvious fix does to an average. Both use the lab’s preset traffic, and every figure except the rule-of-thumb column is exact.

PART ONE — CAPTURE s = 1/100 k = 10 requests missed (99/100)¹⁰ = 0.904382075… 90.44% caught 1 − (99/100)¹⁰ 9.56% expected captures s·k = 0.1 at other rates, same k = 10: s = 1% → 9.56% s = 5% → 40.13% s = 10% → 65.13% s = 25% → 94.37% s = 25.9% → 95.01% first rate to clear 95% PART TWO — THE MEAN traffic 5 ms × 900 8 ms × 60 12 ms × 30 400 ms × 10 true mean = 9 340/1 000 = 9.34 ms keep all ≥ 100 ms, sample the rest at 1%: kept 10 slow requests, 9.9 fast ones total 4 000 + 53.4 = 4 053.4 ms over 19.9 requests naive mean 203.69 ms ×21.8 the truth reweighted: slow ×1, fast ×100 (4 000 + 5 340) / (10 + 990) = 9 340/1 000 = 9.34 ms

The two halves pull in opposite directions and that is the shape of the problem. Uniform sampling gives an honest average and cannot find a rare event; tail-biased sampling finds every rare event and gives a dishonest average. Reweighting is what lets you have both, and it costs nothing except carrying one number per record.

The factor of 21.8 is worth remembering as an order of magnitude rather than a constant: it is roughly the ratio of the tail value to the true mean, scaled by how much of the body was thrown away. Push the sampling rate down and it grows.

For faded practice, take a problem of k = 40 requests at s = 1%. The supplied first move is (99/100)⁴⁰. Compute the capture probability, then find the rate that reaches 95% for this larger k, and say in one sentence why the answer is so much smaller than 25.9% even though the confidence target did not change.

Common mistakes

Reading the sampling rate as the fraction of a problem you will see“One per cent sampling sees one per cent of every problem” is true of totals over millions of requests and false of any individual incident. What a rate gives you for a k-event problem is 1 − (1 − s)ᵏ, and at small s·k that is close to s·k — a tenth of one capture, meaning almost always none at all.

Scaling the rate linearly with the confidence you wantGoing from 9.56% to 95% confidence takes twenty-six times the rate, not ten. The reason is that the confidence is 1 − (1 − s)ᵏ and the quantity that scales cleanly is (1 − s)ᵏ, the probability of missing. Halving the miss probability is what costs a fixed amount of rate, and you need to halve it four times over.

Averaging a deliberately biased sampleThe moment a pipeline keeps “all errors plus 1% of successes” or “everything over 100 ms”, every plain mean and every plain percentile computed downstream describes a population that was never served. The repair is one division per record and it must happen before the aggregation, not after it.

Check your understanding

✓

Finish when a sampling rate is quoted with the capture probability it implies for a stated k.

You should be able to compute 1 − (1 − s)ᵏ, solve it for the rate a confidence requires, say why the answer is nothing like linear, and reweight a non-uniform sample so that its mean is the population mean again.

Note: Keeping everything is the alternative to sampling, and it has its own arithmetic: every distinct combination of label values is a separate series, stored for ever. “Metric Cardinality” multiplies five ordinary labels into 3 456 000 series and 39.81 GB a day, and then adds one more label that multiplies it by ten thousand.