Usability Testing of an Explainable AI-enhanced Tool for Clinical Decision Support: Insights from the Reflexive Thematic Analysis

2025-04-07

AI Summary

Low clinical adoption of AI in healthcare stems primarily from a misalignment between technical explainability and real-world clinical needs. Method: We conducted an empirical, context-sensitive study involving 20 frontline U.S. clinicians, employing qualitative usability testing and reflexive thematic analysis, integrated with human factors engineering and clinical workflow modeling. Contribution/Results: We propose the first structured, actionable operational definition of AI explainability for healthcare, alongside a customizable design framework. This framework systematically characterizes clinicians’ prioritized preferences across three dimensions: *explanatory content* (e.g., salient variables, uncertainty quantification), *presentation modality* (e.g., visualizations, natural-language summaries), and *temporal embedding* (i.e., integration before, during, or after clinical decision-making). By grounding explainability requirements directly in clinical practice, our work bridges the gap between algorithmic transparency and clinical utility, providing an evidence-based foundation and reusable methodology for designing trustworthy, deployable medical AI systems.

Technology Category

• Humans and AI: Explainable AI (XAI) for Human Understanding
• Philosophy and Ethics of AI: Accountability, Interpretability & Explainability
• Machine Learning: Transparent, Interpretable, Explainable ML

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Artificial intelligence-augmented technology represents a considerable opportunity for improving healthcare delivery. Significant progress has been made to demonstrate the value of complex models to enhance clinicians` efficiency in decision-making. However, the clinical adoption of such models is scarce due to multifaceted implementation issues, with the explainability of AI models being among them. One of the substantially documented areas of concern is the unclear AI explainability that negatively influences clinicians` considerations for accepting the complex model. With a usability study engaging 20 U.S.-based clinicians and following the qualitative reflexive thematic analysis, this study develops and presents a concrete framework and an operational definition of explainability. The framework can inform the required customizations and feature developments in AI tools to support clinicians` preferences and enhance their acceptance.

Problem

• Address unclear AI explainability in clinical decision support
• Develop framework for AI explainability to enhance clinician acceptance
• Improve usability of AI tools for healthcare delivery

Innovation

• Explainable AI-enhanced clinical decision support tool
• Usability study with reflexive thematic analysis
• Framework for AI explainability customization