A gauge read every I seconds is a sample of the signal at I-second instants, and a spike of duration d is caught only if a read falls inside it: probability min(1, d/I). At the common fifteen-second interval a three-second spike is seen one time in five, and a spike of up to fourteen seconds can hide completely. The same interval hides a second thing entirely — a counter that restarted, whose only evidence is that the number went down.
Core ideas
A monitoring interval is a sampling rate, and sampling rates have consequences that are arithmetic rather than opinion.
A gauge read at t reports the value at t. Whatever the signal did between t − I and t is not averaged, not maximised and not recorded — it is absent. Two reads fifteen seconds apart both showing a quiet system are two facts about two instants, and the graph drawn between them is a line your plotting library invented.
Reads are I apart, so a spike shorter than I can contain at most one of them, and whether it contains one depends on where it starts. With the start uniform relative to the scrape clock, the chance is the fraction of the cycle the spike occupies: d/I, capped at one when d ≥ I. Three seconds in fifteen is exactly 1/5.
Counters are read as differences, so a process restart that zeroes one turns the next difference into a large negative rate — or, if the counter has already climbed past the previous read, into a small positive one that looks like a quiet spell. The fix is to treat any decrease as a restart and take the new reading as the increment, which recovers the rate and loses whatever accumulated between the last read and the restart.
The material
Where min(1, d/I) comes from and what it assumes, what a run of spikes actually does, and the second thing an interval hides.
The scrape interval I is the period at which a monitoring system reads a target’s current metric values. For a gauge, the value recorded is the one in effect at the instant of the read. For a counter, what is stored is the running total, and a rate is computed as the difference between two reads divided by the time between them.
Let a spike of duration d begin at a time uniformly distributed relative to the scrape clock, and let reads occur every I. If d ≥ I at least one read falls inside the spike and it is caught with certainty. If d < I the spike covers a fraction d/I of the scrape cycle, so
P(caught) = min(1, d/I).
The assumption is the uniformity. It fails exactly when the spike is produced by something on a clock of its own — a cron job, a flush every sixty seconds, or the scrape itself — in which case the phase is fixed and the probability is not d/I but zero or one, permanently.
That failure mode is worth more than the formula. A spike that is always missed is indistinguishable from no spike, and a periodic job whose period shares a factor with the scrape interval produces exactly that. If a graph is suspiciously flat, changing the interval to a value coprime with your suspicions is a cheap experiment.
Fourteen seconds is the figure to carry away. At the default fifteen-second interval that most monitoring is configured with, an outage lasting a quarter of a minute can leave a completely flat graph, and the graph is not lying: it is reporting every value it was given.
The lab draws a run of spikes at seeded times across five minutes and lays the scrape instants over them. At d = 3 and I = 15 the expected fraction caught is a fifth, and the fraction actually caught on a given run is some other number near it — which is what a single afternoon of monitoring is. Reseed and it changes; the probability does not.
This is the same distinction the first page of the course made about a measured ratio, arriving in a different costume. 1/5 is a property of the configuration; “we caught three of twelve yesterday” is a measurement with its own error bar, and twelve spikes is a very small sample.
A service fails hard for twelve seconds and recovers. The scrape interval is fifteen seconds; the outage happens to start two seconds after a read and end one second before the next. Error-rate gauges read zero on both sides. The request counter, being cumulative, does carry the missing successes — a dip in the computed rate that is real and easy to dismiss — but every gauge-based panel and every alert built on one saw a healthy system throughout. Users saw twelve seconds of errors.
Counters only go up, so monitoring systems compute rates by differencing consecutive reads. A process restart sets the counter back to zero, and the restart itself happens between two reads where nothing can observe it. The only evidence is that the second read is smaller than the first, and a naive difference turns that into a spectacular negative rate.
The corrected rate is 34.67 rather than 40 and the gap is exactly the eighty increments that accumulated between the last read and the restart. They are gone: the process that held them no longer exists, and no correction applied downstream can recover a number nobody read. What the correction buys is a rate that is positive, approximately right, and does not poison every aggregate it feeds.
Notice how the error behaves if the restart happens earlier in the window. Once the counter has climbed past its previous reading before the next scrape, the difference is positive again and nothing looks wrong at all — the restart is then completely invisible, and the rate it reports is understated by however much was lost. The large negative number is the friendly case.
Interactive
min(1, d/I), what a flat graph is evidence of, and a counter that went backwards
Method
Start from the event, not from the default. The interval is a consequence of what you are willing to miss.
Ten seconds of elevated errors, a two-second pause, a thirty-second queue build-up. Until this is a number, no interval can be defended, and the default of fifteen seconds is a decision somebody else made about your system.
If it is well below one, you are sampling the event rather than observing it, and the graph will be quiet most of the times it happens. Compute I − 1 as well: that is the longest such event that can leave no trace at all.
A counter accumulates between reads, so a burst of errors during a gap still shows up in the next difference. A gauge samples an instant and a burst between two instants is gone. Where you must use a gauge, a max-since-last-scrape or a histogram of the interval turns it back into something cumulative.
Any rate computation over a counter needs a restart rule. Treat a decrease as a restart and use the new reading as the increment; record that the correction discards whatever accumulated since the previous read, and do not let a negative rate reach an average, a percentile or an alert threshold.
If the event you are hunting is produced by a scheduled job, d/I does not apply: the phase is fixed rather than uniform, and you will either always see it or never see it. Change the interval to something coprime with the job’s period and compare.
Two independent calculations at the same interval. The first is the probability arithmetic, which is exact and needs no data. The second is a single restart, which needs two counter reads and a rule.
Part one is a statement about the configuration and part two is a statement about one event, and they share a cause: everything that happens strictly between two reads is invisible, and only its aftermath is available for inference.
The 34.67 is worth dwelling on. It is not the true rate, it is not a rounding of the true rate, and the difference from 40 is a known quantity — eighty increments over fifteen seconds is 5.33/s, and 34.67 + 5.33 = 40. A correction whose residual you can name is a much better thing to ship than one you cannot.
For faded practice, set the interval to 10 s and the spike to 4 s, and move the restart to 96 s. The supplied first move is that the catch probability becomes 2/5 and the longest invisible spike becomes 9 s. Predict the two counter reads around the restart and both rates before you look, then check, and say which of the two panels changed more when you halved the interval.
You should be able to compute min(1, d/I) and I − 1 for any interval, say what a flat graph is and is not evidence of, state when the uniform-phase assumption fails, and turn two counter reads across a restart into a naive rate, a corrected rate and the increments that are gone.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.