Statistical Foundations of LLM-based A/B Testing: A Surrogacy Framework for Human Causal Inference

2026-06-15

AI Summary

This study addresses the challenge of ensuring that large language models (LLMs), when used as proxies for humans in A/B testing, yield treatment effect estimates that reliably reflect true human causal effects. It introduces surrogate endpoint theory into LLM-based experimentation, establishing a statistical identification framework under conditions weaker than distributional equivalence by calibrating LLM outputs to human responses for average treatment effect estimation. To handle LLM stochasticity, the work proposes a falsifiability principle and a multi-sampling averaging strategy. Additionally, it develops overlap diagnostics and bias bound estimators to assess surrogate validity. Empirical validation on simulated data and the Upworthy headline experiments demonstrates that LLMs can serve as reliable surrogates for historical interventions, though human experimentation remains necessary for evaluating novel interventions.

Abstract

Organizations and researchers show increasing interest in using large language models (LLMs) in place of human participants in A/B tests, in the hope of experimenting faster and at lower cost. We study when a treatment effect estimated on LLM outcomes recovers the effect that would have been measured on the human population of interest. Distributional equivalence between LLM and human outcomes would make any standard estimator valid but is unrealistic. We therefore develop a statistical framework that adapts surrogate endpoint theory to LLMs. The framework shows that calibrating LLM outcomes to human outcomes identifies the average treatment effect under surrogacy and comparability conditions that are jointly weaker than distributional equivalence. When these conditions fail, the effect of interest is only partially identified, and we provide diagnostics that can falsify surrogacy on historical experiments together with a bound on the worst-case bias from limited overlap. We further show that the stochasticity inherent to LLMs introduces both bias and variance, but using an average of multiple draws as the surrogate mitigates both. We illustrate the methods and theory in simulations and an application to A/B tests on Upworthy headlines. A central takeaway from our work is that the validity of LLM outcomes as surrogates can only be falsified for past treatments and never verified for new ones, so human experiments remain indispensable for novel interventions. We discuss the role of LLM choice, prompting, and temperature as design variables, and how to size human experiments for validation.

Problem

• LLM-based A/B testing
• surrogate endpoint
• causal inference
• treatment effect
• human outcomes

Innovation

• surrogacy framework
• LLM-based A/B testing
• causal inference
• distributional calibration
• treatment effect identification