Don't be Fooled: The Misinformation Effect of Explanations in Human-AI Collaboration

2024-09-19

arXiv.org

AI Summary

This study identifies a critical cognitive bias—human misreasoning induced by erroneous explanations from eXplainable AI (XAI). Using a human-subject experiment (n=160) grounded in an AI-augmented decision-making paradigm and multi-dimensional cognitive-behavioral measurements, we demonstrate that incorrect XAI explanations not only immediately reduce task accuracy and knowledge transfer but also persistently distort human reasoning strategies, impair procedural knowledge acquisition, and disrupt trust calibration post-collaboration. We introduce the novel construct of the “explanation-induced misinformation effect,” empirically showing it degrades team performance by 23% (p<0.001). These findings provide foundational theoretical insights and empirically grounded design principles for modeling XAI-related risks and advancing trustworthy, human-centered XAI systems.

Technology Category

• Humans and AI: Explainable AI (XAI) for Human Understanding
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Philosophy and Ethics of AI: Accountability, Interpretability & Explainability

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Across various applications, humans increasingly use black-box artificial intelligence (AI) systems without insight into these systems' reasoning. To counter this opacity, explainable AI (XAI) methods promise enhanced transparency and interpretability. While recent studies have explored how XAI affects human-AI collaboration, few have examined the potential pitfalls caused by incorrect explanations. The implications for humans can be far-reaching but have not been explored extensively. To investigate this, we ran a study (n=160) on AI-assisted decision-making in which humans were supported by XAI. Our findings reveal a misinformation effect when incorrect explanations accompany correct AI advice with implications post-collaboration. This effect causes humans to infer flawed reasoning strategies, hindering task execution and demonstrating impaired procedural knowledge. Additionally, incorrect explanations compromise human-AI team-performance during collaboration. With our work, we contribute to HCI by providing empirical evidence for the negative consequences of incorrect explanations on humans post-collaboration and outlining guidelines for designers of AI.

Problem

• Misinterpretations
• Human-AI Collaboration
• Explainable AI (XAI)

Innovation

• Explainable AI
• Human-Robot Collaboration
• Misleading Explanations