Augmenting Clinical Decision-Making with an Interactive and Interpretable AI Copilot: A Real-World User Study with Clinicians in Nephrology and Obstetrics

2026-01-31

AI Summary

This study addresses clinicians’ distrust of black-box AI systems, which hinders their adoption in high-stakes clinical settings. To overcome this barrier, we propose AICare—an interactive, interpretable AI collaborator that integrates longitudinal electronic health record analysis, explainable visualizations, and large language models to dynamically generate risk predictions and diagnostic suggestions, supporting shared decision-making in nephrology and obstetrics. Our work innovatively establishes a trust mechanism centered on transparent interaction and reveals distinct collaboration strategies across clinician experience levels: junior clinicians use AICare as a cognitive scaffold, while experts employ it for adversarial validation. Empirical evaluations using NASA-TLX, SUS, and qualitative analyses demonstrate that AICare significantly reduces cognitive load, enhances task efficiency and user confidence, and effectively fosters trust while accommodating diverse clinical reasoning styles.

Technology Category

• Philosophy and Ethics of AI: Accountability, Interpretability & Explainability
• Humans and AI: Intelligent User Interfaces
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Clinician skepticism toward opaque AI hinders adoption in high-stakes healthcare. We present AICare, an interactive and interpretable AI copilot for collaborative clinical decision-making. By analyzing longitudinal electronic health records, AICare grounds dynamic risk predictions in scrutable visualizations and LLM-driven diagnostic recommendations. Through a within-subjects counterbalanced study with 16 clinicians across nephrology and obstetrics, we comprehensively evaluated AICare using objective measures (task completion time and error rate), subjective assessments (NASA-TLX, SUS, and confidence ratings), and semi-structured interviews. Our findings indicate AICare's reduced cognitive workload. Beyond performance metrics, qualitative analysis reveals that trust is actively constructed through verification, with interaction strategies diverging by expertise: junior clinicians used the system as cognitive scaffolding to structure their analysis, while experts engaged in adversarial verification to challenge the AI's logic. This work offers design implications for creating AI systems that function as transparent partners, accommodating diverse reasoning styles to augment rather than replace clinical judgment.

Problem

• clinical decision-making
• AI transparency
• clinician trust
• interpretable AI
• healthcare AI adoption

Innovation

• interpretable AI
• interactive AI copilot
• clinical decision support
• trust calibration
• human-AI collaboration