Statistical Inference for Misspecified Contextual Bandits

2026-06-21

AI Summary

This work addresses the challenge of statistical inference in contextual bandits when the outcome model is misspecified, a setting in which existing algorithms like LinUCB can yield non-Gaussian estimators and invalid confidence intervals. The authors propose a Z-estimation framework based on inverse probability weighting that accommodates a broad class of marginal moment objectives. They introduce a novel stability condition—scaled inverse propensity convergence—and establish, for the first time, sufficient conditions guaranteeing consistency and asymptotic normality for a wide range of adaptive policies. By integrating sandwich variance estimation with policy stability theory, the method supports both multi-armed and smooth contextual allocation strategies. Empirical evaluations on synthetic data and the HeartSteps V1 trial demonstrate accurate confidence interval coverage and competitive estimation performance across multiple objectives.

Abstract

Contextual bandit algorithms have transformed modern experimentation by enabling real-time adaptation for personalized treatment. Yet these advantages create challenges for statistical inference due to adaptivity. We study inference with contextual-bandit data without assuming a well-specified outcome model. In this setting, we show a previously overlooked issue: standard algorithms such as LinUCB may fail to stabilize under misspecified working models, leading to non-Gaussian estimator behavior and invalid inference. This issue is practically important, as misspecified working models -- such as approximations of complex dynamical systems -- are often employed by online agents in real-world adaptive experiments to balance reward, computational tractability, and robustness. We develop an inverse-probability-weighted Z-estimation framework for a broad class of marginal moment targets, including projection parameters, structural parameters with noisy contexts, and off-policy values. We identify a stability condition tailored to this framework, scaled inverse-propensity convergence, under which the IPW-Z estimator is consistent and asymptotically normal with a consistent sandwich variance estimator. We further establish sufficient conditions for scaled inverse-propensity convergence for several policy classes, including multi-armed bandit algorithms and smooth contextual allocation policies. Simulations and a HeartSteps V1 real-data-calibrated application show reliable coverage and competitive performance across multiple targets. Overall, our results highlight the importance of stability-aware adaptive design for valid post-experiment inference.

Problem

• contextual bandits
• model misspecification
• statistical inference
• adaptive experiments
• estimator stability

Innovation

• contextual bandits
• model misspecification
• inverse-probability weighting
• asymptotic normality
• adaptive experimentation