An objective of 99.9% successful requests over a month permits (1 − SLO) × requests failures, and at a thousand requests a second that is 2 628 000 of them — the same budget as 43 min 48 s of complete outage. An incident spends duration × failed fraction of it, so a half-broken hour costs half of what a dead hour costs. That last sentence is the whole difference between an objective measured on requests and one measured on the clock.
Core ideas
Three quantities and one conversion. The conversion is where an objective stops being a slogan and starts constraining what may be done this month.
An objective of 99.9% successful requests does not forbid failure; it permits one request in a thousand to fail. Over a month at 1000 requests a second that is 2 628 000 000 requests and a budget of 2 628 000 failures. The budget is a quantity that can be spent, measured, and reported as a fraction consumed, which is what makes it useful.
burn = rate × duration × failed fraction. A thirty-minute total outage at 1000 rps burns 1 800 000 failed requests, which is 68.49% of the month's budget. The same thirty minutes with only a tenth of requests failing burns 180 000, or 6.85%. Two incidents of identical length can differ tenfold in what they cost.
(1 − SLO) × period converts the budget into the length of a complete outage that would spend all of it: 43 min 48 s a month at three nines. That figure is useful for intuition and dangerous as a measurement, because it is only the right cost when every single request is failing. Measure on requests; quote the minutes as an illustration.
The material
What the budget is, how an incident spends it, and why the same outage costs different amounts depending on a number that uptime monitoring cannot see.
A service level objective fixes a target value for an indicator over a period — here, the fraction of requests that succeed. The error budget is the failure the objective permits: (1 − SLO) × N, where N is the number of requests in the period. It is a count, it is fixed when the period begins, and it is spent by incidents until it is gone.
Both lines describe the same permission and the second is the one everybody quotes. Note the period it belongs to: a month here is 2 628 000 seconds, a twelfth of a 365-day year. Under a 30-day convention the same three nines would be 2592 seconds, or 43.2 minutes. Neither is wrong and mixing them is.
The burn is rate × duration × failed fraction, and the third factor is the one that uptime monitoring does not have. A service returning errors to one request in ten is not down by any binary measure, and it is spending budget the whole time. Conversely a two-minute total outage at three in the morning, at a tenth of the daytime rate, costs a tenth of what the same two minutes would cost at noon.
The last column is the budget in minutes at each severity, and the bottom row is the familiar 43.8 — which is the point. The “43.8 minutes a month” figure is the special case of a full outage, and quoting it as though it were the budget itself understates by a factor of ten what a long partial degradation is allowed to cost.
Everything above is computation. What to do when the budget is spent — freeze releases, divert the team to reliability work, revise the objective, accept the overrun — is a policy decision that the arithmetic informs and does not make. The value of separating them is that the argument about policy then happens over an agreed number rather than over whose impression of last month is right.
A thirty-minute total outage at 1000 rps against a three-nines monthly objective burns 1 800 000 of 2 628 000 failures: 68.49% of the month, gone, on day three. 828 000 failures remain, which is another 13.8 minutes of full outage or 138 minutes at ten per cent severity. That is a concrete statement about what the rest of the month can afford, and it is available within minutes of the incident ending.
Set the lab's objective to four nines and leave the incident at thirty minutes: the budget is 262 800 failures and the single incident burns 684.93% of it. A month's budget at four nines is 4 min 22.8 s of full outage, so one half-hour incident overruns the whole month’s budget nearly seven times over. An objective that a single ordinary incident cannot fit inside is not a target, and the arithmetic says so before anyone has to argue about it.
One measurement caution. The budget is defined against the requests in the period, so a traffic-weighted objective forgives failures during quiet hours and punishes them at peak, which may or may not match what users experience. If ten customers each send a thousand requests and one sends a million, a request-weighted objective is mostly about that one customer. The arithmetic here is exact; what it is exact about is a choice that has to be made deliberately.
Interactive
What an incident actually spends, in failed requests
Method
The first step is the one that gets skipped, and skipping it is what turns an objective into an argument.
Successful requests out of total requests, over a calendar month, measured at a named place. Each of those choices changes the number: measured at the load balancer or in the client, counting all endpoints or the ones users care about, a rolling window or a calendar one.
budget = (1 − SLO) × N, where N is the requests in the period. At 1000 rps over a month that is (1/1000) × 2 628 000 000 = 2 628 000 failures. Keep it as a count; converting to minutes too early is what loses the severity.
burn = rate × duration × failed fraction. Use the rate that actually applied during the incident rather than the monthly average, and use the measured failed fraction rather than the fact that an alert fired.
A percentage of the budget consumed, and the remainder expressed both as failures and as the minutes of outage it would buy. Then, separately, the policy question — the arithmetic has nothing to say about that and should not pretend to.
A month with one short total outage and one long partial degradation. Uptime monitoring saw one of them. The budget sees both, and prices the invisible one higher.
The two verdicts disagree completely, and the second one is wrong. Incident B failed one request in seven for six hours and never took the service down, so every binary uptime check passed throughout; it spent more than the entire month's budget on its own. The request-weighted measure sees it because severity is one of its factors and is not one of uptime's.
Notice also that 43 min 48 s appears here as an illustration and is not what was measured. The measurement is 7 440 000 failed requests against 5 256 000 allowed. The minutes are useful for explaining the size of the budget to someone and are the wrong instrument for spending it.
For a faded rehearsal, keep the same objective and rate and suppose the team wants to know how long a 5%-severity degradation may run before it exhausts the month. The supplied first move is that severity enters the burn as a multiplier, so the answer is the budget divided by rate × severity. Compute the minutes, check them against the lab's severity table, and then state what that duration implies about how quickly a partial degradation has to be detected.
You should be able to compute a budget as a count of failures, convert it to an equivalent full outage and say why that conversion is an illustration rather than a measurement, price an incident as rate × duration × severity, and report the share spent without attaching a policy to it.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.