Synthesizing Evidence: Data-Pooling as a Tool for Treatment Selection in Online Experiments

2025-08-14

AI Summary

Online experiments face high estimation variance and poor policy generalizability due to constrained traffic, heterogeneous treatment effects, and overlapping experiment assignments. To address these challenges, we propose the Data-Aggregated Treatment Rollout (DTR) framework—a unified modeling approach that jointly handles both overlapping and non-overlapping traffic scenarios, while accommodating both linear and nonlinear causal models. DTR enables cross-experiment data aggregation and collaborative learning without requiring experimental isolation. Through theoretical analysis, synthetic simulations, and real-world platform evaluations, DTR substantially reduces estimation variance, improves personalized policy accuracy across subpopulations, and enhances cross-experiment deployment coherence. Crucially, DTR maintains robustness under high-dimensional covariates and nonlinear treatment-response relationships—outperforming conventional methods in statistical efficiency and causal decision quality. By maximizing data utility across concurrent experiments, DTR advances scalable, reliable causal inference for large-scale online platforms.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Causality
• Multiagent Systems: Mechanism Design

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Randomized experiments are the gold standard for causal inference but face significant challenges in business applications, including limited traffic allocation, the need for heterogeneous treatment effect estimation, and the complexity of managing overlapping experiments. These factors lead to high variability in treatment effect estimates, making data-driven policy roll out difficult. To address these issues, we introduce the data pooling treatment roll-out (DTR) framework, which enhances policy roll-out by pooling data across experiments rather than focusing narrowly on individual ones. DTR can effectively accommodate both overlapping and non-overlapping traffic scenarios, regardless of linear or nonlinear model specifications. We demonstrate the framework's robustness through a three-pronged validation: (a) theoretical analysis shows that DTR surpasses the traditional difference-in-mean and ordinary least squares methods under non-overlapping experiments, particularly when the number of experiments is large; (b) synthetic simulations confirm its adaptability in complex scenarios with overlapping traffic, rich covariates and nonlinear specifications; and (c) empirical applications to two experimental datasets from real world platforms, demonstrating its effectiveness in guiding customized policy roll-outs for subgroups within a single experiment, as well as in coordinating policy deployments across multiple experiments with overlapping scenarios. By reducing estimation variability to improve decision-making effectiveness, DTR provides a scalable, practical solution for online platforms to better leverage their experimental data in today's increasingly complex business environments.

Problem

• Addresses high variability in treatment effect estimates
• Manages overlapping and non-overlapping traffic scenarios
• Improves decision-making with pooled experimental data

Innovation

• Data pooling across experiments for policy roll-out
• Accommodates overlapping and non-overlapping traffic scenarios
• Robust validation via theory, simulation, and empirical data