Lesson · Measuring Systems · Sizing samples and alerts

Burn-rate Alerts

An objective over a period implies a budget: 99.9% over 30 days is exactly 216/5 = 43.2 minutes of failure. A burn rate b is a multiple of the rate that would spend the budget exactly at the end of the period, so a window w at burn b consumes b·w/period of it and exhausts the budget in period/b. Alerting on the budget rather than on the raw error rate is what makes a fast page and a slow page into two settings of one rule.

Open the interactive lesson Practice

budget = (1 − objective) × period 99.9% of 30 days = 0.001 × 43 200 min = 216/5 = 43.2 minutes exactly 99.9% of a 2 628 000 s month = 43.8 minutes — a different month, also exact a window w at burn b spends b·w/period 14.4× for 1 h → 2% time to exhaustion = period ÷ b 14.4× → 50 hours an SLO is not an SLA: one arithmetic, two consequences

A budget, and the rate it is being spent at

Core ideas

What this lesson establishes

The budget is a quantity of failure you are allowed, the burn rate is how fast you are spending it, and the pair (b, w) is an alert.

An objective is a budget, and the period is part of the definition

99.9% over 30 days permits 0.001 × 43 200 = 43.2 minutes of failure, and nothing about that figure is meaningful without the 30 days. The same objective over a 2 628 000-second month — a twelfth of a 365-day year, which is what “Error Budgets” on the Availability and Failure course uses — permits 43.8 minutes. Both are exact, neither is a rounding of the other, and a budget quoted without its period cannot be compared with anything.

The burn rate is a multiple, not a rate

A burn rate of 1 is the error rate that would spend exactly the whole budget exactly at the end of the period. At a 99.9% objective that is 0.1% of requests failing. A burn of 14.4 means 1.44% of requests failing, and a burn of 0 means none. Expressing it this way makes the arithmetic independent of the objective: b = 14.4 spends 2% of the budget in an hour whether the objective is three nines or five.

A pair (b, w) is an alert, and one pair is never enough

A short window at a high burn catches a severe outage within minutes and is blind to a slow leak; a long window at a low burn catches the leak and arrives hours after a severe outage has spent everything. The standard set is two or four pairs together: 14.4× over 1 hour, 6× over 6 hours, 3× over 1 day, 1× over 3 days — spending 2%, 5%, 10% and 10% of the budget respectively by the time each fires.

The material

The budget, the burn, the exhaustion time, and the pairs

Two conventions for the same objective and which period each assumes, the arithmetic of a window, and why multi-window alerting has the shape it has.

Definition · Error budget and burn rate

For an objective o over a period P, the error budget is (1 − o)·P — the quantity of failure permitted. The burn rate b is the current failure rate divided by 1 − o, so b = 1 spends the budget exactly over P. A window of length w at burn b consumes

b·w/P of the budget, and exhausts it after P/b.

objective 99.9%, period 30 days = 43 200 minutes budget = (1 − 999/1000) × 43 200 = 43 200/1000 = 216/5 = 43.2 minutes exactly the other convention, used by the availability course: a month of 2 628 000 s = 43 800 minutes (a twelfth of 365 days) budget = 43 800/1000 = 43.8 minutes also exactly 43.2 and 43.8 are the same objective over two different months

Say which period you mean, every time. The two figures differ by 1.4% and they show up side by side in postmortems, where one of them gets treated as an error in the other. They are not in conflict: a calendar month has between 40 320 and 44 640 minutes, thirty days is the convention this page uses because it is the one burn-rate alerting is normally quoted against, and a twelfth of a year is the convention that makes twelve months add to one year. Both are defensible; neither is discoverable from the number alone.

“Error Budgets” on the Availability and Failure course also counts the budget in failed requests rather than in minutes, which is a third useful framing and a fourth way for two pages to appear to disagree. A budget is a quantity, a period and a unit, and all three have to travel together.

Theorem · What a window costs and when the budget runs out

At a constant burn b over a window of length w, the failure consumed is b·(1 − o)·w, and as a fraction of the budget (1 − o)·P that is

b·w/P, independent of the objective.

Setting the consumed fraction to 1 and solving for the elapsed time gives exhaustion at P/b. So a burn of 14.4 sustained from now spends the whole 30-day budget in 30/14.4 days, which is 50 hours, and has spent 2% of it after the first hour.

The standard pairs, and why there are several

period 30 days, budget 43.2 min at 99.9% burn window b·w/P of the budget exhausts in 14.4× 1 hour 1/50 2% = 0.864 min 50 hours 6× 6 hours 1/20 5% = 2.16 min 5 days 3× 1 day 1/10 10% = 4.32 min 10 days 1× 3 days 1/10 10% = 4.32 min 30 days

Read the last column and the fourth column together. The fast pair fires having spent 2% of the budget, which is why it can afford to page a human at three in the morning: almost everything is still there. The slow pair fires having spent 10%, over three days, at a burn the fast pair never notices — a burn of 1 is exactly the rate that spends the budget by the end of the period, so it is not an emergency, and it is also not sustainable.

A single pair cannot do both jobs. Choose only the fast one and a service failing 0.15% of requests for a fortnight blows the budget without ever tripping it; choose only the slow one and a total outage pages you after it has already cost a third of the quarter’s allowance. The multi-window rule is not sophistication, it is the minimum number of pairs that covers both failure shapes.

Example · The same incident under two rules

A deploy at 02:00 takes the error rate to 1.44%, a burn of 14.4. Under a raw-rate alert at “more than 1% errors for five minutes” it pages at 02:05 and would equally have paged for a 1.1% blip lasting six minutes that costs 0.02% of the budget.

Under the burn-rate rule it pages at 03:00, having spent 0.864 of the 43.2 minutes, and it does not page for the blip at all. The difference is not the threshold; it is that one rule is about the instantaneous rate and the other is about the quantity of budget the rate is consuming.

An SLO is not an SLA

The arithmetic above is identical for both. What differs is what happens when the last row runs out. An objective is an internal target: missing it spends a budget your own organisation owns, and the usual consequence is a freeze on feature work until it recovers. An agreement is a contract: missing it spends money, because a refund or a credit is attached to it.

Because the consequences differ by an order of magnitude, an agreement is normally set looser than the objective it is measured against — 99.5% promised against 99.9% targeted — so that the internal alarm fires long before the external one does. A team that sets both to the same number has arranged for the first warning and the financial penalty to arrive on the same day.

One more distinction worth keeping: the budget is measured in one unit or the other, minutes or requests, and they are not interchangeable. Forty-three minutes of total unavailability and 0.1% of requests failing all month are both “the whole budget”, and they are very different experiences for a user. Which unit the objective is written in decides which of them you have agreed to.

Interactive

Spending an error budget

What a burn rate costs per window, and when it runs out

Set the objective and the burn

The budget, the share a window spends and the time to exhaustion are exact fractions of a minute. The banner names which period convention the figures assume and what the other one would give, so that two pages of this library quoting 43.2 and 43.8 for the same objective can be reconciled rather than mistrusted. Slide the period to 30 days and the budget reads 216/5 exactly.

The objective

The period (days)30

Burn rate b (tenths)144

Alert window w

Error budget for the period—

Error rate at this burn—

Budget this window spends—

As a share of the budget—

Time to exhaustion—

Budget left when it pages—

A burn rate of 1 spends the whole budget exactly at the end of the period. A burn of 14.4 spends 2% of it in an hour, which is why that pair pages fast; a burn of 1 over three days spends 10% of it, which is why that pair pages slowly and catches the quiet leak the fast one cannot see.

Method

Turning an objective into an alert

The first two steps are definitions, the third is the arithmetic, and the fourth is the one that gets skipped.

State the objective, the period and the unit

99.9% of requests over 30 days is a different promise from 99.9% of minutes over a twelfth of a year, and both are called three nines. Write all three down before computing anything, and put them on the dashboard beside the number.

Compute the budget

(1 − o) × P in whatever unit the objective is written in. For 99.9% over 30 days that is 43.2 minutes; keep it as 216/5 while you are doing arithmetic with it, because the fractions stay exact and the decimals invite rounding.

For each candidate pair, compute b·w/P and P/b

The first says how much of the budget is gone when this pair fires; the second says how long you have if the burn continues. A pair that fires after 30% of the budget is gone is a pair that pages too late, whatever its burn rate looks like.

Take at least one fast pair and one slow pair

Fast catches the outage, slow catches the leak, and neither substitutes for the other. The standard four are 14.4× over an hour, 6× over six hours, 3× over a day and 1× over three days; two of them is a reasonable minimum.

Decide what the page means before it arrives

A burn-rate page says a quantity of budget is being consumed at a rate, not that anything is broken. The response to the fast pair is to stop the bleeding; the response to the slow pair is usually to plan work, and waking someone for it is how a good alert becomes an ignored one.

Budgets, burns and pairs

Score 0 / 0

Choose the answer you can justify.

43.2 minutes, a burn of 14.4, and the hour it takes to spend 2% of it

A 99.9% objective over 30 days, and a deploy that takes the failure rate to 1.44%. Every figure is an exact fraction of a minute; the only decisions in the calculation are the period and the unit.

THE BUDGET objective o = 999/1000 period P = 30 days = 43 200 min budget = (1 − o)·P = (1/1000)(43 200) = 216/5 = 43.2 minutes the other convention: P = 2 628 000 s = 43 800 min budget = 43.8 minutes same objective, different month, both exact THE BURN b = 14.4 means failing at 14.4 × (1 − o) = 14.4/1000 = 1.44% one hour at that burn: fraction b·w/P = 14.4 × 60 / 43 200 = 864/43 200 = 1/50 = 2% minutes 2% of 43.2 = 0.864 minutes time to exhaustion P/b = 43 200/14.4 = 3 000 min = 50 hours THE FOUR PAIRS burn window b·w/P budget spent exhausts 14.4× 1 hour 1/50 0.864 min 50 hours 6× 6 hours 1/20 2.16 min 5 days 3× 1 day 1/10 4.32 min 10 days 1× 3 days 1/10 4.32 min 30 days

The fourth column is why these particular pairs are the standard ones. Every row fires having spent a single-figure percentage of the budget, so every row leaves room to respond; and the rows span burn rates from 14.4 down to 1, so between them they catch a total outage and a leak small enough that nobody would call it an incident.

The bottom row deserves a second look. A burn of exactly 1 is the rate at which the budget is spent precisely on schedule — not an emergency by any instantaneous measure, and exactly the rate that leaves you with nothing at the end of the period. It is the case a raw error-rate threshold can never be tuned to catch, because at a 99.9% objective the raw rate is 0.1% and no threshold distinguishes that from healthy noise in a five-minute window.

For faded practice, take a 99.95% objective over 30 days. The supplied first move is that the budget halves to 21.6 minutes. Compute what the 14.4× pair over one hour now spends as a fraction and in minutes, and say whether the fraction changed — then explain in one sentence why, which is the whole reason burn is expressed as a multiple.

Common mistakes

Alerting on the raw error rate rather than on the budgetA threshold on the instantaneous rate fires for a two-minute blip that costs 0.05% of the budget and stays quiet through a fortnight at 0.15% that spends all of it. The budget framing makes severity and duration one quantity — b·w/P — which is the whole reason for the change of variable.

Quoting a budget without the period it is a budget for43.2 and 43.8 minutes are the same objective over 30 days and over a twelfth of a year, and a budget counted in failed requests is a third quantity again. Two teams comparing numbers without their periods will eventually conclude that one of the two systems is broken, and neither will be.

Running one burn-rate pairA single fast pair is blind to the slow leak that is spending the budget on schedule; a single slow pair arrives after a severe outage has already taken a third of the allowance. The pairs are not alternatives to be chosen between — they cover different shapes of failure, and the minimum useful configuration has one of each.

Check your understanding

✓

Finish when a budget is quoted with its period and its unit, and an alert is a (b, w) pair rather than a threshold.

You should be able to compute (1 − o)·P exactly, convert a failure rate into a burn multiple, compute b·w/P and P/b for any pair, say which of two budget figures assumes which period, and explain why an agreement is set looser than the objective behind it.

Note: A burn-rate rule fires on a quantity that is genuinely being spent. A plain threshold on a noisy ratio fires on the sample, and it fires whether or not anything has changed: “Threshold Alerts and False Alarms” computes that a 2% threshold on 200-request windows at a true 1% error rate trips 5.175% of the time, which is 104.32 pages a week from a perfectly healthy service.