Moving a fraction m of a D-byte dataset at a throttled bandwidth B takes m·D/B, which at the lab's defaults is sixteen hours and forty minutes. That is not idle time: the copy is holding a share of each node's transfer budget, so the node serves fewer requests and its utilisation rises for the whole window — and the wait at the higher utilisation is three times the wait at the lower one. Turning the throttle up shortens the window and makes the utilisation worse, and past a point it puts it above one.
Core ideas
One division for the duration and one subtraction for the capacity, and the two are joined by a dial that moves them in opposite directions.
m · D is the bytes that must move and B is the rate they move at, so the window is their quotient. At m = 25% of 48 TB that is 12.00 TB, and at 200 MB/s it is 60 000 seconds — sixteen hours and forty minutes. A terabyte here is 10¹² bytes and a MB/s is 10⁶ bytes a second, which is how both a disk and a throttle are sold.
A node has a finite transfer budget, and whatever the rebalance takes from it is not available for serving. At 200 MB/s out of 800 MB/s the copy holds a quarter, so a node that served 8 000 /s now serves 6 000 /s. The arrival rate did not change, so utilisation goes from 62.50% to 83.33% and stays there until the copy finishes.
Turn it up and the window shortens and the utilisation worsens; turn it down and the reverse. At 50 MB/s the move takes two days and eighteen hours and forty minutes at ρ = 66.67%; at 200 MB/s it takes sixteen hours and forty minutes at 83.33%; at 400 MB/s it takes eight hours and twenty minutes and the node can no longer serve the arriving load at all. There is no setting that is cheap on both axes.
The material
The duration, the capacity the copy steals, what that does to the wait, and the setting at which shortening the window stops being an option.
A rebalance moves a fraction m of the stored data between nodes, because the shard count changed, because a node was added or lost, or because the placement became uneven. The throttle B is the bandwidth the copy is allowed to use, and it is a deliberate limit rather than a property of the hardware: the whole reason to set one is that the node has other work to do.
Where m comes from is the earlier lesson. Under mod N assignment a growth step moves N/(N+1) of the keys — at seven shards going to eight, 87.50% of the dataset — and on a ring it moves 1/(N+1), or 12.50%. The 25% in the lab's defaults is a middling figure that could come from either a ring on a small fleet or a deliberate re-spread; what this lesson does is turn any such fraction into hours and a utilisation.
The wait figures are not new arithmetic. They are “The M/M/1 Queue” evaluated at two utilisations, and the reason a 21-point rise in ρ triples the wait is the knee that “The Knee: Response Time vs Utilisation” describes: the response time rises without bound as ρ approaches 1, so equal steps in utilisation are not equal steps in latency. A rebalance is a decision to sit further up that curve for a measured number of hours.
The fourth row is the one to look at. Halving the window from sixteen hours to eight takes half the node's transfer budget, which leaves it serving 4 000 /s against an arrival rate of 5 000 /s. That is not a slower system; it is a queue that grows without bound for as long as the copy runs, and “Transient Overload and Draining the Backlog” is what happens afterwards. The throttle has a hard ceiling and it is reached well before the hardware's.
“Moving 12 TB takes about seventeen hours” is the version that gets said in a planning meeting. The version that describes what will actually happen is “for sixteen hours and forty minutes, every node's utilisation is 83.33% instead of 62.50% and the queueing wait is three times normal” — and if there is a latency objective, that is the sentence it has to be checked against. The duration is the easy half of the cost and it is the half that gets reported.
Run the arithmetic on the two fractions from “Rehashing When N Changes” and the argument for consistent hashing stops being theoretical. Growing a seven-shard fleet under mod N moves 7/8 of 48 TB — 42 TB, which at 200 MB/s is 2 d 10 h 20 min at ρ = 83.33% throughout. The same growth step on a ring moves 1/8, or 6 TB, which is eight hours and twenty minutes. The identity 7/8 + 1/8 = 1 is, in the end, a statement about how many days of elevated latency a growth step costs.
It also prices the choice of V from “Virtual Nodes” and the decision to reshard from “Hot Keys and Salting”. A rebalance is the operation that every structural change on this course eventually has to pay for, which is why it is worth computing before the change rather than after it — the duration and the utilisation are both available from numbers you already have.
Interactive
m·D/B of wall clock, and the utilisation the copy holds it at
Method
Compute the duration, then the utilisation, then check the second one against whatever latency objective exists. The order matters because the first number is the one that tempts you to stop.
Under mod N assignment a growth step moves N/(N+1); on a ring it moves about 1/(N+1); a deliberate re-spread moves whatever you choose. The fraction is the largest single lever in this calculation and it is decided by a design choice made much earlier.
m · D / B, in consistent units — a TB is 10¹² bytes and a MB/s is 10⁶ bytes a second. 12 TB at 200 MB/s is 60 000 seconds. Report it in hours, because that is the unit the decision is made in.
The copy takes B/budget of the node's transfer capacity, so the effective service rate is μ(1 − B/budget) and ρ during the move is λ over that. If the result is at or above 1, the throttle is too high and nothing else in the plan matters.
W = 1/(μ − λ) at the reduced service rate. Here it triples, from 0.333 ms to 1.000 ms, and it stays tripled for sixteen hours and forty minutes. That is the sentence the plan needs, and it is the one that decides whether the rebalance happens at this throttle or a lower one.
The window first, then what the node looks like while it is open, then what happens if you try to close it faster.
The last block is where the plan is actually decided. Halving the window is arithmetically available and operationally impossible, because the node cannot serve 5 000 /s on 4 000 /s of remaining capacity. The throttle's ceiling is set by the workload, not by the network.
For a faded rehearsal, keep every other number and take the dataset to 96 TB. The supplied first move is the bytes to move, 24.00 TB. Produce the duration at 200 MB/s, then say what happens to ρ during the move and why, then find the throttle at which the window would be back to sixteen hours and forty minutes — and say whether that throttle is usable at this arrival rate.
You should be able to turn a moved fraction, a dataset size and a throttle into a window, compute the service rate and utilisation the copy leaves, evaluate the wait at both utilisations, and find the throttle at which the node stops keeping up.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.