A closed-loop load generator sends its next request when the last one comes back, so while the system is stalled it sends nothing and the stall is recorded once instead of once per request it delayed. Putting back the requests the schedule called for, each waiting from the moment it was due, moves the p99 of one trace from 2 ms to 990 ms — a factor of 495 — without measuring anything new.
Core ideas
The bias is in the generator, not in the system, and it is invisible in the output because the numbers it reports are all true.
Send a request, wait for the reply, send the next. It is the obvious way to write a load generator and it has a built-in feedback: the slower the system, the less load it receives. During a one-second stall a loop scheduled at 10 ms apart issues no requests at all, so instead of a hundred slow requests the trace contains one, and the other ninety-nine are simply absent from the sample.
Users, upstream services and message queues keep arriving during a stall. The requests the closed loop did not send are requests that would have existed, and each would have waited from the moment it was due until the stall cleared. That is what the correction restores: not an invented distribution, but the schedule the generator itself was configured with.
Every latency the generator recorded stays exactly as recorded. What is added is one entry per request the schedule called for during the stall, at the wait that request would have experienced. The percentile is then the same nearest-rank rule applied to a larger sample — the disagreement between 2 ms and 990 ms is a disagreement about which requests existed, not about what a percentile is.
The material
Why a stall appears once in a closed-loop trace, what the imputed latencies are, and the arrival model each of the two answers assumes.
Coordinated omission is the bias in which a measurement process suspends its own sampling during the events it exists to measure. In a closed-loop load generator, the next request is issued only after the previous reply, so a stall of duration S on a schedule of one request every I suppresses about S/I requests — and every one of them would have been slow.
The name says the important part: the omission is coordinated with the fault. Requests are not dropped at random from the sample, which would be harmless; they are dropped precisely when the system is behaving worst, which is where the percentile you are reporting lives.
Consider a generator configured to send one request every 10 ms with a healthy service time of 2 ms. It sends a hundred requests and during one of them the service stalls for a full second. The trace it produces is ninety-nine requests at 2 ms and one at 1 000 ms.
The naive p99 is 2 ms, which is the healthy service time. That is not a coincidence and it is the signature of the bug: with one slow request in a hundred, rank 99 never reaches it, so the reported 99th percentile is a measurement of the system when nothing was wrong. A report of “p99 = 2 ms” on a run that contained a one-second outage is arithmetically correct and completely uninformative.
The median moves too, and the direction is worth noticing: 2 ms naive against 10 ms corrected. A bias that is usually described as a tail problem reaches the middle of the distribution as soon as the stall is long relative to the schedule.
Imputing the missing requests at their scheduled times is exactly right when arrivals are open: generated by a process indifferent to how long the system is taking. It is exactly wrong when arrivals are genuinely closed — a fixed population of clients that each wait for their own reply before acting again, such as a bounded thread pool or a human at a terminal.
In the closed case the requests really were not made, the users really did wait once, and the naive number is what happened. Applying the correction there double-counts. Deciding which model your workload follows is a modelling judgement the arithmetic cannot make for you, and it is the assumption this page rests on.
The usual first reaction is to raise the generator’s concurrency: a hundred threads instead of one, so a stall blocks only one of them. It moves the number and it does not remove the bias. Each thread is still a closed loop, each still stops issuing during a stall, and a fault that affects the whole service stalls all of them at once — which is the case you are trying to measure. What removes the bias is generating requests on a clock rather than on replies, and recording each request’s latency from its scheduled start rather than from when a thread got round to sending it.
A clock-driven generator issues a request every 10 ms whether or not the last one returned, and timestamps each from its due time. Through the same one-second stall it records a hundred slow requests with waits from 1 000 ms down to 10 ms, and its p99 is 990 ms with no correction applied at all.
The closed-loop generator records one. The corrected sample above is an after-the-fact reconstruction of what the clock-driven generator would have seen, which is why it lands on the same number: the correction is not a heuristic, it is the missing half of the experiment.
A monitoring agent that scrapes its own metrics and blocks while the process is stalled omits exactly the samples that matter. So does a client library that records latency only for requests that completed and drops the ones the user abandoned, and so does any dashboard whose denominator quietly excludes timeouts. In each case the sample is missing the events the measurement exists to detect, and in each case the reported number is a true statement about a sample nobody wanted.
The check is one question: could this measurement have been prevented from being taken by the very fault it is supposed to record? If yes, the number is biased in a direction that flatters, and the size of the bias is what the lab computes.
Interactive
A stall measured once, and the same stall charged to everyone it delayed
Method
Before any arithmetic, decide whether the correction applies at all. Steps two to four are mechanical once it does.
Open means the request rate is set by something outside the system — users, a queue, a cron fleet — and does not slow down when the system does. Closed means a fixed population each waiting on its own reply. The correction belongs to the open case only, and applying it to a closed workload overstates the tail.
One request every I, or whatever rate the generator was configured for. This is the piece of information that makes the correction possible: it is not inferred from the trace, it is what you told the generator to do.
A stall of S starting when a request was due suppresses ⌈S/I⌉ − 1 later requests. The k-th is due k·I into the stall and waits S − k·I until it clears. Add those latencies to the sample; change nothing that was measured.
Same nearest-rank rule, larger n, so the rank moves too. Report both numbers and the ratio, because the ratio is the finding: a factor of 495 is a statement about the measurement apparatus that the corrected number alone does not convey.
The trace is the lab’s preset: a generator issuing one request every 10 ms against a service that normally answers in 2 ms, through a single stall of one second. Every number below comes from that one trace and the schedule it was generated from.
Nothing in the corrected sample was measured twice and nothing was invented. The ninety-nine added entries are the schedule, and the schedule was an input to the experiment. If you object to them you are objecting to the configured request rate, not to the arithmetic.
The factor of 495 is not a general constant. It is S/I scaled by where the ranks land: lengthen the stall and it grows, shorten the scheduled interval and it grows, and take the stall below one interval and it collapses to 1 because no request was due inside it. The lab lets you find the point where the two columns separate.
For faded practice, set the stall to 200 ms and the interval to 50 ms on the same hundred requests. The supplied first move is that three requests were due inside the stall, at 150, 100 and 50 ms of waiting. Predict both the naive and the corrected p99 before you touch the sliders, then check, and say in one sentence why this trace is so much less alarming than the first.
You should be able to recognise a closed-loop trace from its percentiles, count the requests a stall suppressed from the schedule, give each its imputed wait, recompute the percentile on the enlarged sample, and state whether the workload being measured is open or closed.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.