Enough Coin Flips Can Make LLMs Act Bayesian

2025-03-06

AI Summary

This work investigates whether large language models (LLMs) perform Bayesian-style structured reasoning in in-context learning (ICL), or merely rely on surface-level pattern matching. Method: We design a controlled biased-coin-flip task to systematically evaluate, under zero-shot and few-shot settings, (i) prior bias calibration, (ii) contextual evidence weighting, (iii) consistency of posterior updates, and (iv) the role of attention mechanisms—integrating controlled ICL experiments, formal Bayesian modeling, and quantitative prior/posterior analysis. Contribution/Results: We provide the first empirical evidence that LLMs can calibrate priors and approximate Bayesian posterior updates given sufficient in-context examples; their inference biases stem primarily from inaccurate priors—not flawed update mechanisms. Contextual evidence robustly overrides explicit bias instructions, while attention strength exhibits only marginal influence on Bayesian consistency. These findings offer novel insights into the inferential nature of LLMs, distinguishing between prior mis-specification and structural reasoning failure.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large language models (LLMs) exhibit the ability to generalize given few-shot examples in their input prompt, an emergent capability known as in-context learning (ICL). We investigate whether LLMs utilize ICL to perform structured reasoning in ways that are consistent with a Bayesian framework or rely on pattern matching. Using a controlled setting of biased coin flips, we find that: (1) LLMs often possess biased priors, causing initial divergence in zero-shot settings, (2) in-context evidence outweighs explicit bias instructions, (3) LLMs broadly follow Bayesian posterior updates, with deviations primarily due to miscalibrated priors rather than flawed updates, and (4) attention magnitude has negligible effect on Bayesian inference. With sufficient demonstrations of biased coin flips via ICL, LLMs update their priors in a Bayesian manner.

Problem

• LLMs exhibit biased priors in zero-shot settings.
• In-context evidence overrides explicit bias instructions.
• LLMs update priors Bayesianly with sufficient demonstrations.

Innovation

• LLMs update priors using Bayesian methods
• In-context evidence overrides explicit bias instructions
• Attention magnitude minimally affects Bayesian inference