Beyond Satisfaction: From Placebic to Actionable Explanations For Enhanced Understandability

2025-12-06

AI Summary

Current evaluation of system interpretability overrelies on subjective user satisfaction, failing to distinguish vacuous explanations from actionable ones—and thus inadequately capturing how explanations enhance users’ domain understanding. Method: We conduct a controlled user study in the Social Security optimal claiming-age decision task, comparing three conditions: no explanation, vacuous explanation, and actionable explanation. We measure both subjective satisfaction and objective task performance—including mental model accuracy, decision consistency, and counterfactual reasoning correctness. Contribution/Results: Actionable explanations significantly improve mental model accuracy (p < 0.01), yet yield no statistically significant difference in subjective satisfaction compared to vacuous explanations. Only objective behavioral metrics reliably discriminate explanation quality. This work shifts interpretability evaluation from a subjectivity-dominated paradigm toward a principled, mixed-methods framework—integrating subjective feedback with quantifiable task performance—and establishes a new methodological foundation for empirical validation of explainable AI.

Technology Category

• Humans and AI: Explainable AI (XAI) for Human Understanding
• Philosophy and Ethics of AI: Accountability, Interpretability & Explainability
• Computer Vision: Interpretability, Explainability, and Transparency

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Explainable AI (XAI) presents useful tools to facilitate transparency and trustworthiness in machine learning systems. However, current evaluations of system explainability often rely heavily on subjective user surveys, which may not adequately capture the effectiveness of explanations. This paper critiques the overreliance on user satisfaction metrics and explores whether these can differentiate between meaningful (actionable) and vacuous (placebic) explanations. In experiments involving optimal Social Security filing age selection tasks, participants used one of three protocols: no explanations, placebic explanations, and actionable explanations. Participants who received actionable explanations significantly outperformed the other groups in objective measures of their mental model, but users rated placebic and actionable explanations as equally satisfying. This suggests that subjective surveys alone fail to capture whether explanations truly support users in building useful domain understanding. We propose that future evaluations of agent explanation capabilities should integrate objective task performance metrics alongside subjective assessments to more accurately measure explanation quality. The code for this study can be found at https://github.com/Shymkis/social-security-explainer.

Problem

• Critiques overreliance on subjective user satisfaction metrics
• Explores differentiation between meaningful and vacuous explanations
• Proposes integrating objective performance with subjective assessments

Innovation

• Actionable explanations improve objective mental model performance
• Subjective satisfaction fails to differentiate explanation effectiveness
• Integrate objective task metrics with subjective assessments for evaluation