Dynamic Decision-Making under Model Misspecification: A Stochastic Stability Approach

2026-02-19

AI Summary

This study investigates the convergence and regret performance of Thompson sampling in dynamic decision-making under model misspecification. By modeling posterior evolution as a Markov process on the belief simplex, the work provides the first geometric classification of learning behavior in misspecified environments and establishes a unified framework for stochastic stability analysis that bridges Bayesian learning and evolutionary dynamics. Leveraging tools from stochastic stability theory, posterior dynamics, and dimensionality reduction, the analysis identifies three canonical mechanisms governing posterior evolution, precisely characterizing limiting beliefs, action frequencies, and asymptotic regret. The results are further extended to general finite model classes, offering a comprehensive understanding of learning under misspecification.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Bayesian Learning
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Web Mining and Content Analysis: Models for Web evolution
• Economics, Online Markets and Human Computation: Social networks and social learning
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Dynamic decision-making under model uncertainty is central to many economic environments, yet existing bandit and reinforcement learning algorithms rely on the assumption of correct model specification. This paper studies the behavior and performance of one of the most commonly used Bayesian reinforcement learning algorithms, Thompson Sampling (TS), when the model class is misspecified. We first provide a complete dynamic classification of posterior evolution in a misspecified two-armed Gaussian bandit, identifying distinct regimes: correct model concentration, incorrect model concentration, and persistent belief mixing, characterized by the direction of statistical evidence and the model-action mapping. These regimes yield sharp predictions for limiting beliefs, action frequencies, and asymptotic regret. We then extend the analysis to a general finite model class and develop a unified stochastic stability framework that represents posterior evolution as a Markov process on the belief simplex. This approach characterizes two sufficient conditions to classify the ergodic and transient behaviors and provides inductive dimensional reductions of the posterior dynamics. Our results offer the first qualitative and geometric classification of TS under misspecification, bridging Bayesian learning with evolutionary dynamics, and also build the foundations of robust decision-making in structured bandits.

Problem

• model misspecification
• dynamic decision-making
• Thompson Sampling
• Bayesian reinforcement learning
• stochastic stability

Innovation

• model misspecification
• Thompson Sampling
• stochastic stability
• posterior dynamics
• Bayesian reinforcement learning