Deployment and Evaluation of an EHR-integrated, Large Language Model-Powered Tool to Triage Surgical Patients

2026-03-17

AI Summary

This study addresses the limited adoption of evidence-based surgical comanagement (SCM) due to the inefficiency of manual patient screening. To overcome this barrier, we developed and deployed SCM Navigator—the first large language model (LLM) tool integrated directly into an electronic health record (EHR) system—that automatically recommends SCM eligibility by synthesizing preoperative clinical notes, structured data, and clinical rules. A human-in-the-loop workflow was established, wherein clinicians review and validate model recommendations, enabling collaborative triage. Evaluated on 6,193 patients, the system achieved 94% sensitivity and 74% specificity. Post-hoc analysis revealed that most misclassifications stemmed from ambiguities or variations in clinical criteria and workflow practices rather than model deficiencies, thereby demonstrating the safety and feasibility of this AI-assisted triage approach in real-world clinical settings.

Technology Category

• Humans and AI: Planning and Decision Support for Human-Machine Teams
• Planning, Routing, and Scheduling: Model-Based Reasoning
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Surgical co-management (SCM) is an evidence-based model in which hospitalists jointly manage medically complex perioperative patients alongside surgical teams. Despite its clinical and financial value, SCM is limited by the need to manually identify eligible patients. To determine whether SCM triage can be automated, we conducted a prospective, unblinded study at Stanford Health Care in which an LLM-based, electronic health record (EHR)-integrated triage tool (SCM Navigator) provided SCM recommendations followed by physician review. Using pre-operative documentation, structured data, and clinical criteria for perioperative morbidity, SCM Navigator categorized patients as appropriate, not appropriate, or possibly appropriate for SCM. Faculty indicated their clinical judgment and provided free-text feedback when they disagreed. Sensitivity, specificity, positive predictive value, and negative predictive value were measured using physician determinations as a reference. Free-text reasons were thematically categorized, and manual chart review was conducted on all false-negative cases and 30 randomly selected cases from the largest false-positive category. Since deployment, 6,193 cases have been triaged, of which 1,582 (23%) were recommended for hospitalist consultation. SCM Navigator displayed high sensitivity (0.94, 95% CI 0.91-0.96) and moderate specificity (0.74, 95% CI 0.71-0.77). Post-hoc chart review suggested most discrepancies reflect modifiable gaps in clinical criteria, institutional workflow, or physician practice variability rather than LLM misclassification, which accounted for 2 of 19 (11%) false-negative cases. These findings demonstrate that an LLM-powered, EHR-integrated, human-in-the-loop AI system can accurately and safely triage surgical patients for SCM, and that AI-enabled screening tools can augment and potentially automate time-intensive clinical workflows.

Problem

• Surgical co-management
• patient triage
• automated screening
• perioperative care
• clinical workflow

Innovation

• Large Language Model
• EHR-integrated AI
• Surgical Co-Management
• Human-in-the-loop
• Clinical Triage Automation