Robust Sequential Experimental Design for A/B Testing

2026-05-12

AI Summary

This work addresses the vulnerability of existing experimental designs in A/B testing to model misspecification by proposing the first unified robust sequential experimentation framework, applicable to contextual bandits and dynamic environments. Integrating sequential experimental design, robust optimization, and causal inference, the method adaptively optimizes sample allocation under model uncertainty. Theoretical analysis establishes a worst-case upper bound on the mean squared error of treatment effect estimation. Empirical evaluations on both synthetic data and real-world data from a major technology company demonstrate that the proposed approach significantly improves estimation accuracy and robustness compared to existing methods.

Abstract

Experimental design has emerged as a powerful approach for improving the sample efficiency of A/B testing, yet existing designs rely critically on correctly specified models. We study robust sequential experimental design under model misspecification and develop a unified framework that covers both contextual bandit and dynamic settings. Theoretically, we prove that our design bounds the worst-case mean squared error of the estimated treatment effect. Empirically, we demonstrate the effectiveness of the proposed approach using synthetic and real-world datasets from a leading technology company.

Problem

• robust sequential experimental design
• A/B testing
• model misspecification
• treatment effect estimation

Innovation

• robust sequential design
• model misspecification
• A/B testing
• contextual bandits
• treatment effect estimation