🤖 AI Summary
This work addresses the limitations of conventional experimental evaluation methods, which treat multiple metrics in isolation, ignore their interdependencies, and rely on subjective judgments when objectives conflict—hindering scalability. To overcome these challenges, the authors propose the first framework that integrates Bayesian decision theory with hierarchical priors. By designing a custom loss function that incorporates business preferences alongside observed evidence, and leveraging historical experiment data to construct informative priors, the approach enables automated and systematic trade-offs among multiple objectives. Evaluated on both real-world and simulated supply chain experiments at Amazon, the method significantly improves estimation efficiency, streamlines complex decision-making processes, and transcends the constraints of traditional hypothesis testing.
📝 Abstract
Randomized controlled experiments assess new policy impacts on performance metrics to inform launch decisions. Traditional approaches evaluate metrics independently despite correlations, and mixed results (e.g., positive revenue impact, negative customer experience) require manual judgment, hindering scalability. We propose a Bayesian decision-theoretic framework that systematically incorporates multiple objectives and trade-offs by comparing expected risks across decisions. Our approach combines experimenter-defined loss functions with observed evidence, using hierarchical models to leverage historical experiment learnings for prior information on treatment effects. Through real and simulated Amazon supply chain experiments, we demonstrate that compared to null hypothesis statistical testing, our method increases estimation efficiency via informative hierarchical priors and simplifies decision-making by systematically incorporating business preferences and costs for comprehensive, scalable decisions.