Test when the cost of being wrong is low and traffic is sufficient. For strategic product direction, make a judgment call and measure outcomes over a longer horizon.
Strong answers show pragmatism: A/B tests for reversible decisions with measurable outcomes and sufficient traffic, judgment calls for strategic direction, low-traffic features, or irreversible decisions. Best candidates discuss sample size requirements, test duration, and the risk of testing everything rather than making bold product bets.
Tests analytical pragmatism. Over-testing PMs move slowly; under-testing PMs make expensive mistakes. Look for candidates who understand when testing is valuable and when it becomes a crutch for avoiding decisions.