1
A rate is a fraction of trials
The success rate of a treatment is its successes over its trials. Two groups of different sizes can have rates that differ in order from the order of their pooled counts.
Lesson · Science, Induction and Causation · Causation
Correlation, Confounding and Simpson's Paradox
Compare two treatments by the rate of success within each group of patients and then by the pooled rate. In the kidney-stone figures one treatment wins in both groups and loses overall, because it was given to the harder cases. The reversal is arithmetic; which comparison to trust depends on what the groups are.
rate = successes / trials, kept exact
A wins in every group, B wins pooled
the confounder sets who gets which arm
which table to trust is a causal question
When the pooled rate reverses
Core ideas
What this lesson establishes
A correlation is a comparison of rates. The comparison can change sign when a third variable is held fixed, and the lab computes both.
2
A confounder is a variable that changes both
If severity affects who is given a treatment and also affects whether it succeeds, severity confounds the comparison of the treatments.
3
Simpson's reversal is a fact about counts
The pooled comparison reverses when the weights of the groups differ between the two arms. The reversal is a property of the numbers, and nothing is wrong with the arithmetic.
The material
Two treatments, two kinds of stone
The counts first, then the comparison within each group, then the pooled comparison and the variable that explains the gap.
“Reference Classes and Statistical Evidence” showed these counts as two reference classes and left the reason for the reversal to this lesson. A study compared two treatments for kidney stones, open surgery, A, and a keyhole procedure, B. Patients had small stones or large ones, and the study counted successes over patients treated. These are the counts the lab starts with.
stones A: successes / treated B: successes / treated
--------------------------------------------------------
small 81 / 87 234 / 270
large 192 / 263 55 / 80
all 273 / 350 289 / 350
Within small stones A succeeded 81/87, which is 27/29, against 13/15 for B; within large stones A succeeded 192/263 against 11/16. In each group A has the higher rate. Pooled, A has 273/350, which is 39/50, and B has 289/350, and B is higher. The lab's verdict tile reads Reversal.
There is no mistake in either comparison. The reason is in the denominators. A was given to 263 patients with large stones and 87 with small, and B to 80 with large and 270 with small. Large stones are the harder cases, with success rates below those of small stones under both treatments, so A's pooled figure is pulled down by a mix that is mostly hard cases.
Definition · Confounder
A confounder is a variable that influences both which treatment a patient receives and whether the treatment succeeds, so that comparing the treatments without it compares different kinds of patient.
The way to remove the effect of the mix is to give both treatments the same mix. The lab's weight control, switched from pooled to standardised, weights each group by its share of all patients, which is the same weighting for both arms. Under that weighting the lab prints 634983/762700 for A against 6231/8000 for B, about 0.83 against 0.78, so the adjusted comparison agrees with both groups.
Theorem · When the pooled comparison reverses
The pooled comparison disagrees with the comparison in both groups only if the two treatments were given to different proportions of the groups, and the groups themselves differ in their rates. With balanced groups the pooled comparison agrees with the groups.
The second preset shows the balanced case. Both arms have the same number in each group, A wins in each group, and A wins pooled; the lab's verdict is No reversal. The third preset shows the pattern of the first in real counts: two of the largest departments in the 1973 admissions to the University of California at Berkeley. Women were admitted at a higher rate in each department and at a lower rate over the two together, since they applied more to the department that admitted fewer.
Which table to believe is not a question the lab can settle. If severity is the reason the doctors chose a treatment, then comparing within groups is right. If the grouping variable is itself an effect of the treatment, such as a complication that the treatment causes, then conditioning on it removes part of the effect being measured and the pooled comparison is the right one. The same counts can support either answer, and the difference is a claim about what causes what.
That is where this lesson meets the next ones. A correlation between a treatment and an outcome is a comparison of rates; whether it reflects a cause is decided by a model of how the variables influence each other, and “Counterfactual Causation and the But-For Test” is where such models are written down.
Interactive
Rates within groups and pooled
the same counts, compared group by group and all together
Method
Checking a comparison for a reversal
Four steps, in this order, for any comparison of two rates across groups.
1
Compute the rate in each group
For each group and each arm, divide successes by trials and keep the fraction. Say which arm is higher in each group.
2
Compute the pooled rates
Add the successes and the trials over the groups, for each arm separately, and divide. Say which arm is higher.
3
Compare the two verdicts
If the pooled order differs from the order in both groups, there is a reversal. Look at the group sizes in each arm for the reason.
4
Decide what the groups are
If the grouping variable influences who gets which arm and does not result from the arm, trust the within-group comparison. If it results from the arm, trust the pooled one.
Rates, groups and the pooled figure
Score 0 / 0Choose the answer you can justify.
Kidney stones, group by group
Treatment A and treatment B, with small and large stones.
small: A = 81/87 = 27/29, B = 234/270 = 13/15; A higher
large: A = 192/263, B = 55/80 = 11/16; A higher
pooled: A = 273/350, B = 289/350; B higher
A treated 263 large and 87 small; B treated 80 and 270
standardised: A = 634983/762700, B = 6231/8000; A higher
A wins in both groups and loses in the pooled figures, because most of its patients had large stones, which are the harder cases. Weighting both arms the same way removes the effect of the mix and restores A's lead, 634983/762700 against 6231/8000.
Common mistakes
1
Thinking a higher overall rate means a better treatmentIn the kidney counts B has the higher overall rate, 289/350 against 273/350, and A has the higher rate for small stones and for large ones. The overall rate also reflects who was treated, and A treated mostly the hard cases.
2
Always trusting the within-group comparisonComparing within groups is right only when the groups are not themselves caused by the treatment. When they are, the pooled comparison measures what the treatment does, and conditioning on the groups removes some of it.
3
Treating the reversal as a trick of the numbersEach rate is correct, and each comparison answers a different question: how the arms do for a patient of each kind, and how they do for the patients actually treated. The reversal is the sign that the two questions have different answers.
Check your understanding
✓
Finish when you can compute both comparisons, test for a reversal and name the confounder.
Given two tables of counts, you should be able to compute the rate within each group and pooled, say whether the pooled comparison reverses, and name the variable that accounts for it and the facts that decide which comparison to trust.
Note: Nothing here is estimated: the lab compares counts as given and does not say whether a difference could be chance. Statistical inference is left out of the course, and every figure above is a ratio of whole numbers.