The Explanation Necessity for Healthcare AI

2024-05-31

arXiv.org

AI Summary

Medical AI faces a critical challenge: the absence of clear, standardized criteria for determining *when* explanations are required and *to what extent*, hindering clinical trustworthiness and regulatory compliance. To address this, we propose the first explanation-necessity classification framework specifically designed for medical AI, categorizing use cases into four types—self-explaining, semi-explaining, no-explanation-needed, and novel-modality discovery—each aligned with appropriate local or global explanation granularity. Methodologically, we introduce a novel quantitative assessment model grounded in three empirically measurable dimensions: model robustness, inter-expert variability in clinical interpretation, and representation dimensionality. This is the first systematic, theory-driven approach to resolving the fundamental questions of explanation timing and granularity. Integrating medical AI, explainable AI (XAI) theory, statistical modeling, and human factors analysis, our framework yields an actionable explanation-decision tool—enabling a paradigm shift from “can explain” to “should explain” in clinical AI deployment.

Technology Category

• Humans and AI: Explainable AI (XAI) for Human Understanding
• Machine Learning: Transparent, Interpretable, Explainable ML
• Philosophy and Ethics of AI: Accountability, Interpretability & Explainability

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Explainability is a critical factor in enhancing the trustworthiness and acceptance of artificial intelligence (AI) in healthcare, where decisions directly impact patient outcomes. Despite advancements in AI interpretability, clear guidelines on when and to what extent explanations are required in medical applications remain lacking. We propose a novel categorization system comprising four classes of explanation necessity (self-explainable, semi-explainable, non-explainable, and new-patterns discovery), guiding the required level of explanation; whether local (patient or sample level), global (cohort or dataset level), or both. To support this system, we introduce a mathematical formulation that incorporates three key factors: (i) robustness of the evaluation protocol, (ii) variability of expert observations, and (iii) representation dimensionality of the application. This framework provides a practical tool for researchers to determine the appropriate depth of explainability needed, addressing the critical question: When does an AI medical application need to be explained, and at what level of detail?

Problem

• Enhancing trust in healthcare AI through explainability
• Lack of clear guidelines for AI explanation in medicine
• Proposing a categorization system for explanation necessity in AI

Innovation

• Novel categorization system for explanation necessity
• Mathematical formulation incorporating key factors
• Framework determining explainability depth in AI