Robust and efficient multiple-unit switchback experimentation

2025-06-14

AI Summary

When legal, ethical, or engineering constraints preclude user-level randomization, alternatives such as item-level randomization often yield biased causal estimates due to confounding and carryover effects. This paper proposes Regularized Balanced Switch Design (RBSD), the first balanced switch design family that jointly randomizes over time and items while ensuring theoretical unbiasedness and practical robustness. We rigorously characterize sufficient conditions for RBSD to mitigate carryover effects and derive an unbiased estimator within the potential outcomes framework. Extensive simulations and real-world e-commerce experiments demonstrate that RBSD significantly outperforms conventional item-level randomization and unbalanced switch designs: it reduces average estimation error by 32%–57% without introducing additional bias.

Technology Category

• Reasoning under Uncertainty: Causality
• Search and Optimization: Mixed Discrete/Continuous Search
• Multiagent Systems: Mechanism Design

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

User-randomized A/B testing has emerged as the gold standard for online experimentation. However, when this kind of approach is not feasible due to legal, ethical or practical considerations, experimenters have to consider alternatives like item-randomization. Item-randomization is often met with skepticism due to its poor empirical performance. To fill this gap, in this paper we introduce a novel and rich class of experimental designs,"Regular Balanced Switchback Designs"(RBSDs). At their core, RBSDs work by randomly changing treatment assignments over both time and items. After establishing the properties of our designs in a potential outcomes framework, characterizing assumptions and conditions under which corresponding estimators are resilient to the presence of carryover effects, we show empirically via both realistic simulations and real e-commerce data that RBSDs systematically outperform standard item-randomized and non-balanced switchback approaches by yielding much more accurate estimates of the causal effects of interest without incurring any additional bias.

Problem

• Develops robust designs for experiments when user randomization is infeasible
• Addresses poor performance of item-randomized A/B testing alternatives
• Improves causal effect estimation accuracy without introducing bias

Innovation

• Randomly changes treatment over time and items
• Resilient to carryover effects in estimators
• Outperforms standard item-randomized approaches