Policy-Aware Design of Large-Scale Factorial Experiments

2026-04-09

AI Summary

This work addresses the challenge of efficiently identifying optimal strategies in large-scale combinatorial online experiments within shared user populations, where traditional A/B testing is hindered by limited traffic budgets and the combinatorial explosion of factor levels. The authors propose a two-stage centralized factorial experimental design: first, a low-rank tensor model is employed to infer the effects of unobserved combinations and screen critical factor levels; subsequently, sequential halving is applied to the retained combinations to select the best-performing strategy. By reformulating high-dimensional factorial experimentation as a low-rank tensor completion problem, the method eliminates reliance on exhaustive exploration of the full factorial space and enables efficient, selection-oriented experimentation. Experiments on 100 million Taobao user interactions demonstrate that the proposed approach significantly outperforms both one-shot tensor completion and unstructured best-arm baselines, with particularly pronounced advantages under low-budget and high-noise conditions.

Technology Category

• Machine Learning: Online Learning & Bandits
• Search and Optimization: Combinatorial Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Digital firms routinely run many online experiments on shared user populations. When product decisions are compositional, such as combinations of interface elements, flows, messages, or incentives, the number of feasible interventions grows combinatorially, while available traffic remains limited. Overlapping experiments can therefore generate interaction effects that are poorly handled by decentralized A/B testing. We study how to design large-scale factorial experiments when the objective is not to estimate every treatment effect, but to identify a high-performing policy under a fixed experimentation budget. We propose a two-stage design that centralizes overlapping experiments into a single factorial problem and models expected outcomes as a low-rank tensor. In the first stage, the platform samples a subset of intervention combinations, uses tensor completion to infer performance on untested combinations, and eliminates weak factor levels using estimated marginal contributions. In the second stage, it applies sequential halving to the surviving combinations to select a final policy. We establish gap-independent simple-regret bounds and gap-dependent identification guarantees showing that the relevant complexity scales with the degrees of freedom of the low-rank tensor and the separation structure across factor levels, rather than the full factorial size. In an offline evaluation based on a product-bundling problem constructed from 100 million Taobao interactions, the proposed method substantially outperforms one-shot tensor completion and unstructured best-arm benchmarks, especially in low-budget and high-noise settings. These results show how centralized, policy-aware experimentation can make combinatorial product design operationally feasible at platform scale.

Problem

• factorial experiments
• combinatorial interventions
• policy selection
• experimentation budget
• interaction effects

Innovation

• factorial experiments
• low-rank tensor completion
• policy-aware design
• sequential halving
• combinatorial optimization