Simpson’s paradox
Sample Ratio Mismatch
Referrals vs. online applications
Sample Ratio Mismatch
Referrals vs. online applications
Mathematically, there is really no paradox. Consider the following table:
| Treatment A | Treatment B |
|---|---|
| a/b | A/B |
| c/d | C/D |
| \(\frac{a+c}{b+d}\) | \(\frac{A+C}{B+D}\) |
It is completely possible to have the following relationships:
\[ \begin{align} a/b &< A/B \\ c/d &< C/D \\ \frac{a+c}{b+d} &> \frac{A+C}{B+D} \end{align} \] This is what the Simpson’s paradox looks like mathematically, however, conceptually, it becomes trickier to explain, and that’s why it is called a Paradox. Let’s see an example.
Assume the above scenario, where we have two potential treatments, A and B, that can potentially cure a disease. Assume that the disease can be severe and not severe. If we don’t properly randomize the populations who will take the drugs, then maybe we will end up in a situation where a majority of severe patients end up in one group. For instance:
| Treatment A | Treatment B | |
|---|---|---|
| Not Severe | 9/10 | 85/100 |
| Severe | 60/100 | 5/10 |
| Aggregate | 69/110 | 90/110 |
So in the above, 9/10 > 85/100 and 60/100 > 5/10 but once combined: 69/110 < 90/110 and hence we have the Simpson’s paradox.
Note that in order for Simpson’s paradox to occur, there needs to be unequal data splits (i.e., a result of buggy randomization that generates selection bias). For instance, in the above example, the Simpson’s paradox disappears if the groups are evenly split, e.g.:
| Treatment A | Treatment B | |
|---|---|---|
| Not Severe | 90/100 | 85/100 |
| Severe | 60/100 | 50/100 |
| Aggregate | 150/200 | 135/200 |
The Simpson’s paradox is an extremely popular topic in interviews, and it often arises in subtle ways (e.g., combining experiments with different results). It is extremely important that you understand this topic otherwise it might be an easy-to-miss topic in an interview setting.
Simpson’s paradox does not show up exclusively in A/B testing questions. For instance, it can come up in Machine Learning system design sessions (p.186 Huyen (2022)), as it can occur when comparing different models in production (e.g., model B might be overall better than model A, but model A might perform better on each individual subgroup of the population).
Simpson’s paradox
- Normality assumption Medium (Normality)
- Sample Ratio Mismatch Medium (Sample Ratio Mismatch)
- Reducing variance in AB testing Medium (Variance)
- Multi-armed and contextual bandits in AB testing Medium (Contextual bandits, Multi-armed bandits)
- False discovery control Easy (False discovery rate, Multiple hypotheses testing, Benjamini & Hochberg, Bonferroni)
Thank you, we have received your feedback.