You can access a significantly larger sample of the platform's content for free by logging in with your Gmail account. Sign in now to explore.

What is Simpson’s paradox and can you give an example where it can arise?

. The Simpson’s paradox is when a relationship that appears in different groups of data disappears or even reverses when the groups are combined. It typically occurs when there is a confounding variable that affects the distributions of individual groups.

Mathematically, there is really no paradox. Consider the following table:

Treatment A | Treatment B |
---|---|

a/b | A/B |

c/d | C/D |

\(\frac{a+c}{b+d}\) | \(\frac{A+C}{B+D}\) |

It is completely possible to have the following relationships:

\[ \begin{align} a/b &< A/B \\ c/d &< C/D \\ \frac{a+c}{b+d} &> \frac{A+C}{B+D} \end{align} \] This is what the Simpson’s paradox looks like mathematically, however, conceptually, it becomes trickier to explain, and that’s why it is called a Paradox. Let’s see an example.

Assume the above scenario, where we have two potential treatments, A and B, that can potentially cure a disease. Assume that the disease can be severe and not severe. If we don’t properly randomize the populations who will take the drugs, then maybe we will end up in a situation where a majority of severe patients end up in one group. For instance:

Treatment A | Treatment B | |
---|---|---|

Not Severe | 9/10 | 85/100 |

Severe | 60/100 | 5/10 |

Aggregate | 69/110 | 90/110 |

So in the above, 9/10 > 85/100 and 60/100 > 5/10 **but once combined**: 69/110 < 90/110 and hence we have the Simpson’s paradox.

Note that in order for Simpson’s paradox to occur, there needs to be unequal data splits (i.e., a result of buggy randomization that generates selection bias). For instance, in the above example, the Simpson’s paradox disappears if the groups are evenly split, e.g.:

Treatment A | Treatment B | |
---|---|---|

Not Severe | 90/100 | 85/100 |

Severe | 60/100 | 50/100 |

Aggregate | 150/200 | 135/200 |

The Simpson’s paradox is an extremely popular topic in interviews, and it often arises in subtle ways (e.g., combining experiments with different results). It is extremely important that you understand this topic otherwise it might be an easy-to-miss topic in an interview setting.

Simpson’s paradox does not show up exclusively in A/B testing questions. For instance, it can come up in Machine Learning system design sessions (p.186 Huyen (2022)), as it can occur when comparing different models in production (e.g., model B might be overall better than model A, but model A might perform better on each individual subgroup of the population).

You can find more on Simpson’s paradox here: https://en.wikipedia.org/wiki/Simpson%27s_paradox

Simpson’s paradox

- Sample Ratio Mismatch Medium (Sample Ratio Mismatch)
- Reducing variance in AB testing Medium (Variance)
- Equal-sized treatment and control groups Medium (Power, Variance, Sample size)
- False discovery control Easy (False discovery rate, Multiple hypotheses testing, Benjamini & Hochberg, Bonferroni)
- Multi-armed and contextual bandits in AB testing Medium (Contextual bandits, Multi-armed bandits)