An estimate cannot be checked by redoing it. It is checked by reaching the same quantity along a route that shares no assumption with the first, and then asking whether the two answers agree — where “agree” means inside the interval the inputs earned, not “close”. The failure mode is specific and common: a second route that borrows a factor from the first cannot contradict it, because a shared factor cancels out of the ratio between them.
Core ideas
One hard idea — independence is what makes a check a check — stated three ways: what a second route has to avoid, what agreement has to mean, and what a shared factor does to the arithmetic.
Route A goes from people: uploaders, photos each, bytes each. Route B goes from the object store: objects written, mean size measured on disk. They arrive at the same quantity — bytes uploaded per day — through different data, and that is the only property that makes the second one informative. Redoing the same multiplication more carefully is not a check; it is the same claim, twice.
2.0·10¹³ against 1.8·10¹³ is a ratio of 10/9. Whether that counts as agreement depends entirely on how well the inputs were known: against the ÷8 … ×8 band that three ±2× factors earn, it is comfortable agreement; against factors known to a few percent it would be a serious disagreement. The band comes from “Orders of Magnitude”, and without it the word agree means nothing.
Give route B the same 2 MB per photo that route A assumed, and the two answers become 2.0·10¹³ and 2.0·10¹³ and the ratio becomes exactly 1 — for every possible value of that factor. A quantity that appears in both products divides out of their quotient, so no value of it could ever make the check fail. An agreement nothing could have broken has tested nothing.
The material
Why a second route is the only available check, what agreement has to mean, and how a shared factor makes the whole exercise empty.
To triangulate an estimate is to compute the same quantity by a second route whose factors are independent of the first — no factor appears in both chains — and to compare the two answers by their ratio. The estimates agree when the ratio lies within the band ÷k … ×k that the inputs’ half-widths earn.
The reason a second route is the only check available is that nothing else can fail. The arithmetic can be verified by redoing it, and the units can be verified by writing them down, and neither of those can tell you that a factor was wrong. An assumption is only testable against evidence that did not come from it.
The two routes share no factor. Route A rests on a guess about how many photos a person uploads and on an assumed size per photo; route B rests on an object count and a mean size measured on disk. Drag the assumed size per photo in the lab and route A moves while route B stays exactly where the measurement put it, which is precisely what makes the comparison capable of failing.
Now give route B the assumed 2 MB per photo instead of the measured 1.8 MB. Both routes now say 2.0·10¹³ bytes a day, the ratio is exactly 1, and the check looks better than it did. It is worth nothing: the shared factor divides out of the quotient, so the ratio stays at 1 whatever value that factor takes. The lab marks the shared row in red and reports the verdict as “no check made”, which is the accurate description.
A fake check does not look like a failure; it looks like an unusually good result. That is what makes it worth a lesson. The test is mechanical rather than intuitive: list the factors of both chains and look for a name appearing twice. If one does, the agreement carries no information about it, and the honest move is to say which factor remains untested rather than to report the ratio.
The lab makes the mechanism visible by letting you drag the shared factor. With independent routes the ratio moves and the check can fail; with the factor shared the ratio does not move at all, however far the slider goes. A quantity no possible value of which could change the outcome has not been tested.
The band is the half-width of the estimate, computed in “Orders of Magnitude”: three factors known to ±2× give ÷8 … ×8. Two routes whose answers differ by 10/9 are well inside that and are telling the same story to the accuracy anyone claimed. Two routes differing by 20× are not, and no amount of explaining will make them.
So the band has to be fixed before the two answers are compared, and it has to come from the inputs rather than from the results. Widening k after seeing a disagreement is the same defect as a shared factor, arriving a step later: it makes the check unfailable after the fact.
A disagreement outside the band is the useful outcome, not the failed one. It means one of the assumptions is wrong, and it usually says which: the factors the two routes do not share are the suspects, and the one the two routes disagree about most is the place to go and measure. Rechecking the multiplication is the wrong response, because the multiplication was never the part carrying the uncertainty.
One last piece of procedure, which is a fact about the order you do things in rather than about arithmetic. Build the second route before looking at the first answer again. A route constructed with the target number in mind will reach it, by a series of small and individually defensible choices, and the resulting agreement will be as empty as a shared factor and much harder to see.
Interactive
Two routes, one ratio, and the factor that would make it meaningless
Method
Fix the band, build a route that could disagree, compare last.
It comes from the half-widths of the first route’s factors: three at ±2× give ÷8 … ×8. Fixing k in advance is what stops the comparison from being adjusted to fit whatever comes out, which is the most comfortable way to fail this lesson.
Different data, different people, a different part of the system. Do not consult the first answer while doing it — a chain built toward a known number reaches it, one defensible choice at a time, and the agreement means nothing afterwards.
Mechanical, and quicker than reasoning about it. Any factor common to both products cancels out of their ratio, so it cannot be contradicted by the comparison. If one shows up, say so and name it as untested rather than reporting the ratio as a check.
A ÷ B, against ÷k … ×k. Inside the band the routes agree to the accuracy the inputs claimed; outside it they do not, and the band does not get widened at this point.
The multiplication is not where the uncertainty was. Compare the factors the two routes do not share, find the one they most disagree about, and measure that. A disagreement that identifies a factor is the most useful thing an estimate can produce.
Two routes to one quantity, the ratio between them, and then the same comparison with one factor shared.
The last block is the one to sit with. The ratio did not merely come out well, it came out at exactly 1, and it will come out at exactly 1 for every possible value of the shared factor. Nothing about that comparison could have gone wrong, which is the same as saying nothing about it was a test.
The independent version can fail, and that is its entire value. Move the assumed bytes per photo in the lab and route A moves while route B stays put; push it far enough and the ratio leaves the band, at which point you have learnt that the assumption is incompatible with what the object store measured.
For a faded rehearsal, take the lab’s requests-a-day example: daily active users times sessions each times requests per session, against sessions counted in the access log times requests per session counted in the log. The supplied first move is to fix the band before computing anything. Then decide, before you look at the verdict, whether those two routes are independent — the phrase “requests per session” appears in both, and the question is whether the two figures came from the same place.
You should be able to build an independent second route to a quantity you have estimated, fix the agreement band from the half-widths before comparing, detect a shared factor by inspection of the two chains, and say what a disagreement identifies rather than what it invalidates.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.