13 Real A/B Testing Interview Questions and Answers

-- For Data Scientists, Applied Scientists, and Product Data Scientists --

This is a curated, evolving list of real A/B testing interview questions and answers, designed by a former Meta Staff Data Scientist who is still actively interviewing candidates (albeit currently as a Staff ML Scientist). Practicing these questions will help you prepare for Data Scientist and Product Data Scientist roles at FAANG and similar-tier companies.

Problem	Topics	Difficulty
Sample Ratio Mismatch	Sample Ratio Mismatch	Medium
( Login required ) Normality assumption	Normality	Medium
( Login required ) Reducing variance in AB testing	Variance	Medium
( Login required ) Simpson’s paradox	Simpson’s paradox	Medium
( Subscription required ) AA tests	Variance	Easy
( Subscription required ) Counterfactual definition	Counterfactual	Easy
( Subscription required ) Equal-sized treatment and control groups	Power, Variance, Sample size	Medium
( Subscription required ) False discovery control	False discovery rate, Multiple hypotheses testing, Benjamini & Hochberg, Bonferroni	Easy
( Subscription required ) Interference	Interference	Easy
( Subscription required ) Multi-armed and contextual bandits in AB testing	Contextual bandits, Multi-armed bandits	Medium
( Subscription required ) Novelty and primacy effects	Novelty effects, Primacy effects	Easy
( Subscription required ) Randomization checks	Randomization	Easy
( Subscription required ) Randomization level	Randomization, Variance	Medium