LLMs Should Not Yet Be Credited with Decision Explanation

2026-05-01

AI Summary

This study addresses the common conflation in current research between large language models’ (LLMs) predictions and post hoc rationalizations with genuine explanations of human decision-making, which risks overattributing explanatory power to these models. The work distinguishes among three types of claims—decision prediction, reason generation, and causal explanation—and introduces the “explanatory credit bridge criterion” and the “credit calibration principle.” Drawing on theories of explanation from philosophy and cognitive science, it employs behavioral modeling and empirical validation to assess whether LLMs satisfy key explanatory desiderata, including target specificity, discriminability among alternatives, and sensitivity to interventions. By clarifying the appropriate role of LLMs in modeling human decisions, this research establishes a rigorous evaluative framework that guards against mistaking predictive narratives for causal explanations.

Abstract

This position paper argues that LLMs should not yet be credited with decision explanation. This matters because recent work increasingly treats accurate behavioral prediction, plausible rationales, and outcome-conditioned reasoning traces as evidence that LLMs explain why people decide as they do, risking a premature redefinition of what counts as explanatory progress in human decision modeling. We first distinguish three claims with different evidential burdens: decision prediction, rationale generation, and decision explanation. We then argue that the evidence most commonly offered for LLM-based decision accounts directly supports the first two claims, and sometimes explanatory hypothesis generation, but does not distinguish decision explanation from prediction-supportive rationalization. Next, we propose a bridge standard for decision-explanation credit: stronger claims should specify explanatory targets, discriminate against weaker rationalizer alternatives, use target-appropriate process- or intervention-sensitive validation, and bound their scope. We then situate this standard against competing views and related literatures, clarifying why it preserves the value of LLMs as predictors, narrators, and hypothesis generators while resisting premature explanatory credit. We conclude with a principle of credit calibration: LLMs should be credited for the strongest claim their evidence warrants, and no stronger; if adopted, this principle can help turn LLMs from persuasive narrators of decisions into more reliable instruments for discovering, testing, and communicating explanations of human behavior.

Problem

• decision explanation
• large language models
• rationalization
• explanatory credit
• behavioral prediction

Innovation

• decision explanation
• large language models
• rationalization
• explanatory validation
• credit calibration