Storage is not a rate times a time. It is a sum of a rate over a window, which happens to equal rate times time only while the rate is constant. Let the ingest grow and the sum becomes geometric, and the answer pulls away from the product. Then the replication factor multiplies everything, and the three-line calculation that everyone does in their head has two missing factors in it.
Core ideas
One idea with three consequences: a window is a sum of terms, the terms are equal only when nothing is growing, and the copies multiply the sum rather than the rate.
What is on disk after R days is the total of what arrived on each of those days. While the daily ingest is constant, all the terms are equal and the sum collapses to rate × days — an arithmetic series with common difference zero, which is why the product ever works. It is a special case, and treating it as the definition is what breaks the moment anything grows.
At 5% a month the ingest is 1.05ᵐ times its starting value in month m, so the window is a run of monthly blocks each at its own constant rate and the total is a geometric sum. Over ninety days the blocks add 15 TB, 15.75 TB and 16.54 TB for a total of 47.29 TB, against 45 TB if the ingest had stayed flat. The last block runs at 551.25 GB a day, 1.103× where it started.
Three copies of 47.29 TB is 141.86 TB. The copies multiply the accumulated total rather than the ingest rate, which gives the same answer here and stops giving it as soon as anything else in the chain is not linear. Keeping replication as the final step also keeps the unique-data figure visible, and that is the figure a retention policy is actually about.
The material
Why the answer is a series, what growth does to it, and the two factors the quick calculation leaves out.
The retention window R is how long data is kept before it is deleted. Steady state is reached once the system is older than R: each day’s arrivals are matched by one day’s deletions, and the total held stops growing for that reason and starts growing again for any other.
So the quantity to size is what accumulates across R days of arrivals. Write iₘ for the ingest per day during block m; the unique data held is Σ iₘ · (days in block m), and everything that follows is about what the terms of that sum look like.
If the ingest never changes, every term is the same and the sum is rate × days. That is an arithmetic series with common difference zero, and it is the only case in which storage is a product. It is also the case everyone has in mind when they multiply, which is why the product feels like a definition rather than a special case.
An ingest growing at g a month is multiplied by (1 + g) once per month, so during month m — counting from m = 0 — the daily ingest is i₀ · (1 + g)ᵐ. The window is a run of 30-day blocks, each with its own constant rate, and the total is a geometric sum. Growth here steps once a month rather than compounding continuously, which keeps every term an exact fraction.
Each block’s rate is an exact power: month m runs at i₀ · (21/20)ᵐ, with both ends integers, so the lab never prints a figure it had to round on the way. The table in the lab is the series term by term, and the total underneath it is the sum — which is worth watching once, because it is the only way to see that the curve pulls away from the straight line gradually rather than at a point.
The unique data is 47.29 TB and the storage is 141.86 TB. The quick calculation — half a terabyte times ninety days — gives 45 TB, which is short by the growth and short again by the copies. Both factors are recoverable: the growth costs 1.0508× against the flat case and the copies cost exactly 3×.
Over ninety days at 5% a month the growth is worth 1.0508×, which is small enough to feel like a rounding. That feeling is about the window, not about growth. Take the lab’s fast-growth example — 100 GB a day, kept two years, growing 15% a month — and the same arithmetic gives 581.13 TB of unique data against 73 TB if it had stayed flat. Growth is worth 7.9607× there, and the daily ingest in the last month has reached 2.86 TB.
The reason is the one the two prerequisite algebra courses make: a geometric sum is dominated by its last terms, and how much it is dominated by them depends on how many terms there are. Ninety days is three terms and two years is twenty-five, which is the whole difference between a 5% correction and a factor of eight.
The growth here steps once a month and is flat within the month, which is a modelling choice and is visible in the lab as a staircase rather than a curve. It keeps every term exact, and it means a figure read off for day forty-five is the second block’s rate rather than something interpolated. Nothing on this course needs finer resolution than that, and the alternative — a continuously compounded day — would make every term irrational for no gain in what the estimate can tell you.
The units are decimal throughout: a kilobyte is 1000 bytes and a terabyte is 10¹², which is what a disk is sold in. Mixing in 1024-based units partway through a chain produces an error that grows with every thousand-step and looks like nothing at all, because both numbers are plausible.
Interactive
An arithmetic series at constant ingest, a geometric one under growth
Method
A rate per day, a window, a growth rate, a replication factor, in that order.
Usually this is the write rate from “Read/Write Ratio” times the bytes a write carries, converted from per-second to per-day. Do the conversion once, at the start, so the retention window can be in days and the arithmetic can be a sum of daily terms.
Ninety days at monthly growth is three blocks of thirty. Within a block the rate is constant, so each term is a rate times a count of days; across blocks the rate is multiplied by (1+g) each time.
The sum is what one copy of the data occupies, and it is the number a retention policy is about. Write it down separately before multiplying by anything, because it is the figure you will be asked to justify.
Copies multiply the accumulated total. Three copies of 47.29 TB is 141.86 TB; the replication factor is a count of complete copies and belongs at the end of the chain, where it is visible and easy to change.
rate × days × copies is the estimate somebody else will do in their head, so quoting the ratio between your answer and theirs — 1.0508× here, from growth alone — is the fastest way to explain where the difference came from.
Three monthly blocks, one term each, then the copies. Every figure below is in the lab under the same settings.
The three block totals are not equal, and that is the entire difference between this and rate × days. They are also not very unequal, because ninety days is only three terms — which is why growth looks like a rounding error at short windows and stops looking like one at long ones.
Switch the lab to its fast-growth example to see the other end of that: 100 GB a day kept two years at 15% a month is twenty-five blocks, 581.13 TB of unique data, 1.74 PB across three copies, and a growth factor of 7.9607× against the flat answer. Same arithmetic, same three inputs, an answer that is nothing like a product.
For a faded rehearsal, take 2 TB a day kept a year with no growth and two copies. The supplied first move: with g = 0 every term is equal, so the sum really is rate × days. Predict the unique data and the total, then check them in the lab — and then turn the growth up to 5% and say, before you look, roughly what factor the twelve blocks will cost you.
You should be able to turn an ingest rate, a retention window, a monthly growth rate and a replication factor into a storage figure, state the unique-data total separately from the replicated one, and say what factor the growth was worth and why that factor depends on the length of the window.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.