🤖 AI Summary
This study addresses the frequent conflict between surrogate metrics and core North Star metrics in online experimentation, where a principled framework for their integrated decision-making has been lacking. The authors propose a dynamic optimal fusion method that establishes, for the first time, a formal trade-off framework unifying metric quality and experimental power within a single decision system: it prioritizes the North Star metric when experimental power increases, while upweighting the surrogate metric when its validity is high. The approach estimates optimal fusion weights from historical experiment data by jointly leveraging statistical power analysis and surrogate metric validity assessment. Deployed on Netflix’s experimentation platform, this method significantly enhances decision efficiency and the precision of resource allocation in online experiments.
📝 Abstract
Proxy metrics are widely used to improve the precision and velocity of online experimentation (aka A/B testing). Although proxies are often motivated by long-term outcomes that the experimenter does not observe, in many settings they are used alongside a contemporaneous but statistically insensitive north star. This can lead to a practical dilemma: when should experimenters trust the proxy metric, and when should they trust the north star? In this paper, I propose an optimal blending approach that smoothly guides decision-making towards the north star as the power of the experiment increases and away from the north star as the quality of the proxy metric improves. I study the implications of this decision-making framework for the design of experiments and of experimentation programs. Equipped with better (worse) proxy metrics, experimenters should run smaller and more (larger and fewer) experiments. I show how to leverage past experiments to estimate optimal blending weights and experiment sizes. Lastly, I describe the real-world application of the methodology to an experimentation program at Netflix.