Human-Centered Development of an Explainable AI Framework for Real-Time Surgical Risk Surveillance

2025-04-03

AI Summary

There is an urgent clinical need for real-time, interpretable, and low-burden surgical risk prediction tools during the perioperative period. Method: This study introduces a clinician–engineer co-design paradigm to develop MySurgeryRisk—a system integrated into real-world surgical workflows that automatically predicts nine postoperative complications in real time without manual input and provides clinically intelligible explanations. Clinicians were deeply engaged throughout requirements elicitation, cognitive modeling, interface design, and iterative validation via 11 focus groups (20 surgeons/anesthesiologists), thematic coding, and prototype evaluations—yielding five core clinical requirement themes. Contribution/Results: Empirical evaluation demonstrates high feasibility, strong clinical acceptability, and a clear implementation pathway. MySurgeryRisk establishes a methodological framework and practical exemplar for embedding trustworthy AI in high-stakes clinical settings.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Humans and AI: Planning and Decision Support for Human-Machine Teams
• Reasoning under Uncertainty: Applications

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Background: Artificial Intelligence (AI) clinical decision support (CDS) systems have the potential to augment surgical risk assessments, but successful adoption depends on an understanding of end-user needs and current workflows. This study reports the initial co-design of MySurgeryRisk, an AI CDS tool to predict the risk of nine post-operative complications in surgical patients. Methods: Semi-structured focus groups and interviews were held as co-design sessions with perioperative physicians at a tertiary academic hospital in the Southeastern United States. Participants were read a surgical vignette and asked questions to elicit an understanding of their current decision-making practices before being introduced to the MySurgeryRisk prototype web interface. They were asked to provide feedback on the user interface and system features. Session transcripts were qualitatively coded, after which thematic analysis took place. Results: Data saturation was reached after 20 surgeons and anesthesiologists from varying career stages participated across 11 co-design sessions. Thematic analysis resulted in five themes: (1) decision-making cognitive processes, (2) current approach to decision-making, (3) future approach to decision-making with MySurgeryRisk, (4) feedback on current MySurgeryRisk prototype, and (5) trustworthy considerations. Conclusion: Clinical providers perceived MySurgeryRisk as a promising CDS tool that factors in a large volume of data and is computed in real-time without any need for manual input. Participants provided feedback on the design of the interface and imaged applications of the tool in the clinical workflow. However, its successful implementation will depend on its actionability and explainability of model outputs, integration into current electronic systems, and calibration of trust among end-users.

Problem

• Develop an explainable AI framework for real-time surgical risk surveillance
• Understand end-user needs for AI clinical decision support in surgery
• Improve actionability and explainability of AI model outputs for surgeons

Innovation

• Co-designing AI with clinical end-users
• Real-time surgical risk prediction
• Explainable AI for clinical decisions