Do LLMs Act Like Rational Agents? Measuring Belief Coherence in Probabilistic Decision Making

2026-02-06

AI Summary

This study investigates whether large language models (LLMs) exhibit the belief consistency and preference stability expected of rational agents in high-stakes decision-making—specifically, whether their probabilistic judgments and action choices align with Bayesian expected utility maximization. To this end, we develop a falsifiable evaluation framework grounded in Bayesian decision theory and assess multiple LLMs on medical diagnosis tasks that require coherent alignment between reported beliefs and chosen actions. Our findings reveal that LLMs frequently violate fundamental coherence conditions required of rational agents, demonstrating systematic deviations from normative rationality under uncertainty. This work establishes a novel paradigm for evaluating LLM rationality by systematically applying formal rationality criteria to real-world, high-risk decision scenarios for the first time.

Technology Category

• Reasoning under Uncertainty: Decision/Utility Theory
• Knowledge Representation and Reasoning: Reasoning with Beliefs
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) are increasingly deployed as agents in high-stakes domains where optimal actions depend on both uncertainty about the world and consideration of utilities of different outcomes, yet their decision logic remains difficult to interpret. We study whether LLMs are rational utility maximizers with coherent beliefs and stable preferences. We consider behaviors of models for diagnosis challenge problems. The results provide insights about the relationship of LLM inferences to ideal Bayesian utility maximization for elicited probabilities and observed actions. Our approach provides falsifiable conditions under which the reported probabilities \emph{cannot} correspond to the true beliefs of any rational agent. We apply this methodology to multiple medical diagnostic domains with evaluations across several LLMs. We discuss implications of the results and directions forward for uses of LLMs in guiding high-stakes decisions.

Problem

• Large Language Models
• Rational Agents
• Belief Coherence
• Probabilistic Decision Making
• Utility Maximization

Innovation

• belief coherence
• rational agents
• probabilistic decision making
• Bayesian utility maximization
• large language models