A metric with labels is not one time series; it is one series per distinct combination of label values, so the count is the product of the cardinalities. Five ordinary labels give 3 456 000 series, which at a fifteen-second scrape and two bytes a sample is 39.81 GB a day. Adding a label is not adding a column — it is multiplying, and a label with ten thousand values multiplies the whole bill by ten thousand.
Core ideas
One product, one multiplication, and the operational consequence that the arithmetic does not show.
http_requests_total{service, endpoint, status, instance, method} is not a metric with five columns. It is a family, and every distinct tuple of values that has ever been observed is its own time series with its own retained history. The count is the product of the cardinalities, and a product of five modest numbers is not a modest number.
Each series is written once per scrape, so the daily sample count per series is 86 400/I. Multiply by the bytes a compressed sample occupies — a couple of bytes in a modern format, because timestamps and values in a slowly changing series compress extremely well — and the total falls out. Every one of the three factors is a decision somebody made.
The product counts combinations that exist; the index counts combinations that have ever existed. An instance label that changes on every deploy adds a fresh set of series each release and the old ones remain in the index until they age out of retention. This is called churn, it is invisible in the product above, and it is the usual reason a metrics system that was sized correctly falls over three months later.
The material
How a series count is computed, what it turns into per day, and which labels multiply a system out of existence.
A time series is identified by a metric name together with a complete set of label values. The cardinality of a label is the number of distinct values it takes. For a metric with labels of cardinality c₁, …, c_m, the number of series is at most
∏ cᵢ = c₁ × c₂ × … × c_m,
reached when every combination occurs. Real systems reach some fraction of it, and the product is the number to plan against.
Nothing in that list is unreasonable on its own. Forty services is a medium company, a hundred and twenty endpoints is one large service, six status classes is fewer than HTTP defines, thirty instances is a small fleet and four methods is restrained. The product of five reasonable numbers is three and a half million series and forty gigabytes a day, for one metric family.
Now add a sixth label. A region and zone pair at fifty combinations multiplies everything by fifty: 172.8 million series and about 2 TB a day, which is a large and survivable number. A customer label at five hundred multiplies by five hundred. A user_id label at ten thousand multiplies by ten thousand.
The last row is the mistake that gets made, and it is worth being precise about why it is qualitatively different rather than merely larger. A request_id is unbounded: its cardinality is not a hundred thousand or a million, it is however many requests you serve, growing without limit, and every one of those series is written exactly once and then kept for the retention period. A metric with an unbounded label is not an expensive metric; it is a log with the worst possible storage engine.
endpoint="/users/12345/orders" looks like one label with a sensible name, and its cardinality is the number of users. The same metric with endpoint="/users/{id}/orders" has cardinality one for that route. The difference between the two is a templating step in the instrumentation, and it is the difference between a hundred and twenty series and ten million.
The tell is that the cardinality of a label should be predictable from the design of the system rather than from its traffic. If you cannot state a label’s cardinality without querying production, it is probably unbounded.
The bytes on disk are the easy part; the index is not. Every series needs an entry mapping its label set to its data, that index is consulted on every query and is usually held in memory, and a query that touches a million series has to merge a million streams before it can sum them. A dashboard panel that was fast at three thousand series is not slow at three million — it times out.
This is also why histogram resolution and cardinality are the same budget. Every bucket boundary is a label value, so the choice made in “Histograms and Bucket Error” — a ratio of 2 against a ratio of 5/4, at 3.1 times as many buckets — multiplies this product by 3.1 as well. Precision in the tail and the number of labels you can afford are drawn from one account.
And compute the product before shipping the label, not after. The multiplication takes ten seconds and the alternative is discovering the answer from an out-of-memory kill on the storage layer at a time of somebody else’s choosing.
Interactive
The product of the cardinalities, and the gigabytes a day it turns into
Method
Four multiplications and one question. The question is the one that prevents the outage.
Including the ones the client library adds for you: instance, job, pod, and whatever your service mesh injects. A cardinality you did not choose still multiplies.
That is the series count. Compare it to what your storage currently holds; a single metric family that is a large fraction of the whole system’s series count is a finding on its own.
series × (86 400/I) × bytes per sample. Then multiply by the retention in days for the number that actually gets provisioned, and remember that the index is separate and is in memory.
A label whose value comes from a user, a request, a session, a URL or an error message is unbounded regardless of what it looks like today. Unbounded is not a big number; it is a different kind of number, and no capacity plan covers it.
One metric family, five labels nobody would query in review, and a fifteen-second scrape. The arithmetic is a product and three multiplications, and the only reason it surprises anyone is that the labels are added one at a time over eighteen months.
Halving the scrape interval to 7.5 seconds doubles the last block and changes nothing about the first: the sample rate is a storage lever and the labels are a series lever, and only the series lever touches the index. When a metrics system is in trouble it is almost always the series count rather than the byte count, which is why the first question is always the product.
Notice also what the sixth label does to the four-label decisions above it. Dropping method entirely divides everything by four and saves 30 GB a day; it also removes the ability to separate reads from writes. Adding user_id multiplies by ten thousand and answers a question that a trace store answers better. The two decisions are not in the same unit of importance and they are usually made with the same amount of thought.
For faded practice, keep the five labels, set the scrape interval to 30 seconds and add the region × zone label at 50. The supplied first move is that the series count is 3 456 000 × 50. Compute the new daily bytes, then say which of the two changes — the halved sample rate or the fiftyfold label — your storage layer will notice first, and why.
You should be able to compute a series count as a product, turn it into bytes a day from a scrape interval and a sample size, say what one more label of a given cardinality does to both, and identify a label whose cardinality is set by traffic rather than by design.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.