Concurrency is not a number anyone chooses: it is λW, a throughput multiplied by a latency. Fix any two of L, λ and W and the third follows, which makes the same identity a pool sizer, a throughput ceiling and a latency budget depending on which one you leave blank.
Core ideas
Nothing new is proved here. What is new is which of the three letters you leave unknown, and that choice is the whole of capacity sizing.
A service handling λ requests a second, each spending W seconds inside, has λW of them inside at any moment whether or not anyone planned for it. A pool smaller than that number does not reduce the concurrency; it converts the excess into queueing in front of the pool.
Read the same identity the other way: N connections, each occupied for W, can start at most N/W pieces of work per second. A pool of 20 at 40 ms caps at 500 a second, and no amount of offered load gets more than that through it.
If W doubles and N is fixed, the ceiling halves. That is why a dependency getting slower shows up as a throughput incident rather than a latency one: the pool was the binding constraint, and N/W moved.
The material
The three rearrangements, the unit trap in the middle of them, and the thread-count question they settle.
With L the mean number in flight, λ the arrival rate and W the mean time in system: L = λW sizes the concurrency, λ = L/W gives the throughput a fixed concurrency supports, and W = L/λ gives the time in system implied by the other two. They are one equation rearranged, in the sense of Algebra's literal-equations-and-formulas.
The only difficulty is units. λ is per second and W is usually quoted in milliseconds, so the product needs one conversion — and skipping it produces an answer a thousand times too large, which is large enough to be noticed and small enough to be believed once someone has typed it into a configuration file.
The pool of 20 above supports 500 a second at 40 ms. If the database slows to 80 ms, the same pool supports 20/0.08 = 250 a second. Holding 500 a second through it now needs 500 × 0.08 = 40 connections. Nothing about the traffic changed.
A thread that is waiting on a network call is not using a core. Sizing a thread pool by core count is right for work that is CPU-bound and wrong for work that is not, and the identity says by how much: a service taking 200 requests a second, each spending 250 ms mostly waiting on a dependency, has 200 × 0.25 = 50 requests in flight. Eight cores is the wrong number to compare that with; 50 is the concurrency the work has, and the question is only whether the runtime can hold 50 waiting things cheaply.
This is also the honest way to read a “max connections” limit on a database. It is not a popularity cap; it is a W-dependent throughput cap, and the number it implies changes every time the query mix changes.
One warning before the lab. Sizing a pool at exactly L sizes it at the mean, and the mean is not the peak: arrivals bunch, and a pool sized at the mean is a queue in front of a queue. How much headroom that argues for is what the rest of this course prices, and the answer is a function of how close to its ceiling you are willing to run.
Interactive
Concurrency, throughput and latency: fix any two
Method
Name the unknown first. Everything else is one multiplication or one division.
A pool size is L, a ceiling is λ, an implied latency is W. Writing down which one is unknown before touching numbers prevents the answer coming out upside down.
Milliseconds divided by 1000. Do it explicitly and leave it written down: 40 ms = 0.04 s.
L = λW, λ = L/W, W = L/λ. One operation; if the arithmetic is longer than that, a unit was left in.
A pool much larger than λ in seconds' worth of requests, or an answer in the tens of thousands for a service that handles hundreds a second, is the millisecond-times-per-second error.
The number the identity gives is a mean. Whatever multiple of it you choose, record the multiple as a decision rather than rolling it into the estimate.
One sizing, then the same pool re-read as a ceiling when the dependency gets slower.
The third block is the useful one. A pool is a throughput cap expressed in the units of whatever latency the dependency happens to have today, so a latency regression and a throughput regression are the same event seen through N/W.
For a faded rehearsal, size a pool for 2000 requests a second at 5 ms, then state the ceiling that same pool imposes if the service time doubles. The supplied first move is 5 ms = 0.005 s; predict both numbers before opening the quiz.
You should be able to size a pool from a rate and a latency, state the ceiling any fixed pool implies, convert units without thinking about it, and explain why the core count is the wrong comparison for work that waits.
Something to change about this lesson? Write it down, tick it off when it is handled, and export the list when you want to act on it. Kept in this browser only: nothing is sent anywhere, and nobody is notified.
No recommendations yet for this lesson.