Evaluating Decision Rules Across Many Weak Experiments

2025-02-12

AI Summary

In large-scale, low-signal A/B testing at technology companies, existing decision rules lack unbiased and comparable evaluation criteria, leading to severe bias in estimating cumulative North Star metric gains under low signal-to-noise ratios. To address this, we propose the first cross-validation–based unbiased estimator for cumulative treatment effects, integrating causal inference principles with meta-analytic thinking to overcome the high bias inherent in conventional plug-in estimators in weak experiments. We further develop a quantitative evaluation framework for decision rules centered on cumulative North Star metric gains. Evaluated on 123 historical experiments at Netflix, our rule increases cumulative gains by 33% and has been deployed in production. Our core contributions are: (i) the first unbiased estimator for cumulative treatment effects in online experimentation, and (ii) a novel, industrially grounded paradigm for benchmarking and comparing decision rules on a common, interpretable metric.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Sequential Decision Making
• Multiagent Systems: Mechanism Design

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Technology firms conduct randomized controlled experiments ("A/B tests") to learn which actions to take to improve business outcomes. In firms with mature experimentation platforms, experimentation programs can consist of many thousands of tests. To effectively scale experimentation, firms rely on decision rules: standard operating procedures for mapping the results of an experiment to a choice of treatment arm to launch to the general user population. Despite the critical role of decision rules in translating experimentation into business decisions, rigorous guidance on how to evaluate and choose decision rules is scarce. This paper proposes to evaluate decision rules based on their cumulative returns to business north star metrics. Although intuitive and easy to explain to decision-makers, this quantity can be difficult to estimate, especially when experiments have weak signal-to-noise ratios. We develop a cross-validation estimator that is much less biased than the naive plug-in estimator under conditions realistic to digital experimentation. We demonstrate the efficacy of our approach via a case study of 123 historical A/B tests at Netflix, where we used it to show that a new decision rule would have increased cumulative returns to the north star metric by an estimated $33%$, directly leading to the adoption of the new rule.

Problem

• Evaluating decision rules for scaling A/B testing programs
• Estimating cumulative returns of decision rules with weak signals
• Improving business outcomes through optimized experiment decision rules

Innovation

• Cross-validation estimator reduces bias significantly
• Evaluates decision rules via cumulative returns
• Applied to 123 Netflix A/B tests successfully