Contrastive explanations of BDI agents

2026-02-10

AI Summary

This work proposes the first integration of contrastive explanations—such as “Why was action X performed rather than Y?”—into the Belief-Desire-Intention (BDI) agent architecture to enhance transparency and user trust. By extending the agent’s reasoning mechanism, the system can automatically generate concise contrastive explanations. Evaluation through both computational metrics and human-subject studies demonstrates that this approach significantly reduces explanation length, with some evidence indicating higher user preference. Moreover, it effectively improves users’ understanding of the agent’s behavior, strengthens trust, and increases confidence in the system’s correctness. However, the study also reveals a nuanced finding: in certain contexts, providing more complete explanations may paradoxically diminish user trust, highlighting the importance of explanation design tailored to specific interaction scenarios.

Technology Category

• Knowledge Representation and Reasoning: Reasoning with Beliefs
• Humans and AI: Explainable AI (XAI) for Human Understanding
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

The ability of autonomous systems to provide explanations is important for supporting transparency and aiding the development of (appropriate) trust. Prior work has defined a mechanism for Belief-Desire-Intention (BDI) agents to be able to answer questions of the form ``why did you do action $X$?''. However, we know that we ask \emph{contrastive} questions (``why did you do $X$ \emph{instead of} $F$?''). We therefore extend previous work to be able to answer such questions. A computational evaluation shows that using contrastive questions yields a significant reduction in explanation length. A human subject evaluation was conducted to assess whether such contrastive answers are preferred, and how well they support trust development and transparency. We found some evidence for contrastive answers being preferred, and some evidence that they led to higher trust, perceived understanding, and confidence in the system's correctness. We also evaluated the benefit of providing explanations at all. Surprisingly, there was not a clear benefit, and in some situations we found evidence that providing a (full) explanation was worse than not providing any explanation.

Problem

• contrastive explanations
• BDI agents
• transparency
• trust
• autonomous systems

Innovation

• contrastive explanations
• BDI agents
• explainable AI
• trust development
• human-subject evaluation