When Does Interference Matter? Decision-Making in Platform Experiments

2024-10-09

AI Summary

This paper investigates interference effects in A/B testing on inventory-constrained online platforms and their impact on the decision quality of frequentist hypothesis tests. Addressing cross-unit interference induced by inventory constraints under customer-level randomization, the authors develop a Markov chain model to theoretically analyze A/A test behavior and the bias–variance trade-off. Their contributions are threefold: (1) They prove, for the first time, that the naive difference-in-means estimator strictly controls Type I error under monotonic interventions—without requiring debiasing; (2) they demonstrate that this estimator achieves superior statistical power over existing debiased methods in large-state-space regimes; and (3) they show that its performance degrades severely under non-monotonic interventions, and accordingly propose a practical criterion to guide whether debiasing is necessary. The results provide both theoretical foundations and actionable guidelines for experiment design in resource-constrained platform environments.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Online Learning & Bandits
• Multiagent Systems: Mechanism Design

Application Category

• Economics, Online Markets and Human Computation: Economics and fairness of platforms and recommendation systems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Security and Privacy: Large-scale security measurements

Abstract

This paper investigates decision-making in A/B experiments for online platforms and marketplaces. In such settings, due to constraints on inventory, A/B experiments typically lead to biased estimators because of interference; this phenomenon has been well studied in recent literature. By contrast, there has been relatively little discussion of the impact of interference on decision-making. In this paper, we analyze a benchmark Markovian model of an inventory-constrained platform, where arriving customers book listings that are limited in supply; our analysis builds on a self-contained analysis of general A/B experiments for Markov chains. We focus on the commonly used frequentist hypothesis testing approach for making launch decisions based on data from customer-randomized experiments, and we study the impact of interference on (1) false positive probability and (2) statistical power. We obtain three main findings. First, we show that for monotone treatments -- i.e., those where the treatment changes booking probabilities in the same direction relative to control in all states -- the false positive probability of the na""ive difference-in-means estimator with classical variance estimation is correctly controlled. This result stems from a novel analysis of A/A experiments with arbitrary dependence structures. Second, we demonstrate that for monotone treatments, when the state space is large, the statistical power of this na""ive approach is higher than that of any similar pipeline using a debiased estimator. Taken together, these two findings suggest that platforms may be better off not debiasing when treatments are monotone. Finally, we numerically investigate false positive probability and statistical power when treatments are non-monotone, and we show that the performance of the na""ive approach can be arbitrarily worse than a debiased approach in such cases.

Problem

• Analyzes interference impact on decision-making in platform A/B experiments
• Examines false positive probability and statistical power in experiments
• Compares naive vs debiased estimators for monotone and non-monotone treatments

Innovation

• Analyzes Markovian model for inventory-constrained platforms
• Focuses on frequentist hypothesis testing for launch decisions
• Compares naive vs debiased estimators for interference impact