When Medical AI Explanations Help and When They Harm

2025-12-09

AI Summary

This study identifies a “transparency paradox” in medical AI explanations: explanations improve diagnostic accuracy by 6.3 percentage points when AI predictions are correct, yet reduce accuracy by 4.9 percentage points when AI errs. Based on a behavioral experiment involving 3,855 diagnostic decisions by 257 medical students—and integrating econometric analysis with welfare-effect modeling—we find that current explanation methods lack discriminative credibility calibration, leading clinicians to overtrust erroneous AI recommendations. To address this, we propose a “selective transparency” strategy—providing explanations only when AI confidence exceeds a calibrated threshold. Relative to mandatory full transparency, this approach generates 43% higher healthcare value, yielding an estimated annual benefit of $2.59 billion. Our core contribution is the first empirical identification of a conditional reversal mechanism in explanation utility—where explanations shift from beneficial to harmful depending on AI correctness—and the derivation of a practical, welfare-optimized governance framework for clinical AI deployment.

Technology Category

• Machine Learning: Transparent, Interpretable, Explainable ML
• Humans and AI: Explainable AI (XAI) for Human Understanding
• Philosophy and Ethics of AI: Accountability, Interpretability & Explainability

Application Category

• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Security and Privacy: Data transparency and provenance
• Responsible Web: Algorithmic accountability and transparency on the web

Abstract

We document a fundamental paradox in AI transparency: explanations improve decisions when algorithms are correct but systematically worsen them when algorithms err. In an experiment with 257 medical students making 3,855 diagnostic decisions, we find explanations increase accuracy by 6.3 percentage points when AI is correct (73% of cases) but decrease it by 4.9 points when incorrect (27% of cases). This asymmetry arises because modern AI systems generate equally persuasive explanations regardless of recommendation quality-physicians cannot distinguish helpful from misleading guidance. We show physicians treat explained AI as 15.2 percentage points more accurate than reality, with over-reliance persisting even for erroneous recommendations. Competent physicians with appropriate uncertainty suffer most from the AI transparency paradox (-12.4pp when AI errs), while overconfident novices benefit most (+9.9pp net). Welfare analysis reveals that selective transparency generates $2.59 billion in annual healthcare value, 43% more than the $1.82 billion from mandated universal transparency.

Problem

• Explanations improve decisions when AI is correct but worsen them when AI errs.
• Physicians cannot distinguish helpful from misleading AI explanations.
• Selective transparency generates more healthcare value than universal transparency.

Innovation

• Selective transparency based on AI correctness
• Persuasive explanations regardless of recommendation quality
• Welfare analysis favoring selective over universal transparency