A Definition of Good Explanations and the Challenges Explaining LLM Outputs

2026-06-12

AI Summary

This work addresses the challenge that current large language models (LLMs) face in generating intelligible explanations, largely due to the absence of a clear definition of what constitutes a “good” explanation. To bridge this gap, the paper proposes a novel definition that integrates counterfactual explanations with the interlocutor’s prior beliefs, thereby linking explanation quality directly to the recipient’s cognitive state and extending traditional counterfactual frameworks. Grounded in philosophical and cognitive science theories, this approach systematically elucidates the root causes of LLMs’ explanatory difficulties and identifies the core components of effective explanations. The resulting framework offers both theoretical foundations and practical guidance for developing more comprehensible and user-aligned explainable AI systems.

Abstract

How to define a good explanation is a long-standing philosophical debate which has found recent renewed interest in the context of AI outputs. Explainability is crucial for AI adoption in many contexts, but in order to produce good explanations of AI systems, we must first have an understanding of what good explanations are. In this paper we propose a definition inspired by the notion of counterfactual explanations, however we argue that one must also take into account the interlocutor's prior beliefs in each fact that could be offered in an explanation. We explore the ramifications of this definition for AI explainability and, in particular, why LLM outputs are difficult to produce good explanations for.

Problem

• explanation
• explainability
• large language models
• counterfactual
• prior beliefs

Innovation

• counterfactual explanations
• prior beliefs
• AI explainability
• LLM interpretability
• explanation quality