MUSE: Multi-Treatment Experiment Design for Winner Selection and Effect Estimation

2025-10-06

AI Summary

This paper addresses the two-stage experimental practice—first selecting the best treatment, then estimating its effect—in multi-arm trials (e.g., clinical trials, A/B/n tests). We propose the first experimental design framework jointly optimizing both selection accuracy of the winning treatment and precision of its effect estimation. Methodologically, we unify these objectives into a single optimization criterion, leveraging Neyman’s allocation principle; a theory-driven algorithm determines optimal sample allocations across the control and all treatment arms. The framework provides finite-sample theoretical guarantees and achieves asymptotic optimality. Simulation studies and real-data experiments demonstrate that our approach significantly outperforms conventional balanced allocation and sequential designs in both winning-treatment identification accuracy and average treatment effect estimation precision.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Multiagent Systems: Mechanism Design
• Game Theory and Economic Paradigms: Mechanism Design

Application Category

• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

We study the design of experiments with multiple treatment levels, a setting common in clinical trials and online A/B/n testing. Unlike single-treatment studies, practical analyses of multi-treatment experiments typically first select a winning treatment, and then only estimate the effect therein. Motivated by this analysis paradigm, we propose a design for MUlti-treatment experiments that jointly maximizes the accuracy of winner Selection and effect Estimation (MUSE). Explicitly, we introduce a single objective that balances selection and estimation, and determine the unit allocation to treatments and control by optimizing this objective. Theoretically, we establish finite-sample guarantees and asymptotic equivalence between our proposal and the Neyman allocation for the true optimal treatment and control. Across simulations and a real data application, our method performs favorably in both selection and estimation compared to various standard alternatives.

Problem

• Designing experiments for multiple treatment levels selection
• Jointly maximizing winner selection and effect estimation accuracy
• Optimizing unit allocation between treatments and control groups

Innovation

• Jointly optimizes winner selection and effect estimation
• Introduces single objective balancing selection and estimation
• Determines unit allocation by optimizing combined objective