🤖 AI Summary
In online A/B testing, joint inference across multiple metrics faces a fundamental trade-off: metric aggregation incurs information loss, while stringent multiple-testing correction severely diminishes statistical power. This paper proposes the first Bayesian experimental design framework that jointly optimizes false discovery rate (FDR) control and average statistical power. Innovatively formulating FDR and power as dual constraints, our method efficiently computes the optimal sample size and decision thresholds via only two numerical simulations—bypassing the computational bottleneck of traditional intensive Monte Carlo simulation. Evaluated on real-world multi-metric scenarios, the approach achieves strict FDR control at ≤0.1 while improving average statistical power by 22% relative to standard methods. Moreover, it accelerates computation by over 90% compared to exhaustive grid search, enabling scalable, principled multi-metric experimentation.
📝 Abstract
Online controlled experiments (i.e., A/B tests) are a critical tool used by businesses with digital operations to optimize their products and services. These experiments routinely track information related to various business metrics, each of which summarizes a different aspect of how users interact with an online platform. Although multiple metrics are commonly tracked, this information is often not well utilized; multiple metrics are often aggregated into a single composite measure, losing valuable information, or strict family-wise error rate adjustments are imposed, leading to reduced power. In this paper, we propose an economical framework to design Bayesian A/B tests while controlling both power and the false discovery rate (FDR). Selecting optimal decision thresholds to control power and the FDR typically relies on intensive simulation at each sample size considered. Our framework efficiently recommends optimal sample sizes and decision thresholds for Bayesian A/B tests that satisfy criteria for the FDR and average power. Our approach is efficient because we leverage new theoretical results to obtain these recommendations using simulations conducted at only two sample sizes. Our methodology is illustrated using an example based on a real A/B test involving several metrics.
