AI Agents for Conversational Patient Triage: Preliminary Simulation-Based Evaluation with Real-World EHR Data
🤖 AI Summary
Current medical dialogue AI systems lack high-fidelity, clinically verifiable patient simulation environments. Method: We propose the first electronic health record (EHR)-driven patient simulator, which automatically extracts structured clinical vignettes from real EHRs and integrates clinical knowledge modeling with an LLM-driven, multi-turn symptom elicitation mechanism to generate dynamic triage dialogues covering diverse symptoms and diagnoses. Contribution/Results: We establish the first data-driven, clinically calibrated simulation benchmark for evaluating conversational triage AI. Dual expert blind evaluation confirms 97.7% behavioral consistency between simulated patients and original cases, and 99% semantic correlation between generated clinical summaries and ground-truth EHR documentation. This framework significantly enhances ecological validity, clinical credibility, and reproducibility in training and evaluating dialogue-based triage systems.
Application Category
📝 Abstract
Background: We present a Patient Simulator that leverages real world patient encounters which cover a broad range of conditions and symptoms to provide synthetic test subjects for development and testing of healthcare agentic models. The simulator provides a realistic approach to patient presentation and multi-turn conversation with a symptom-checking agent. Objectives: (1) To construct and instantiate a Patient Simulator to train and test an AI health agent, based on patient vignettes derived from real EHR data. (2) To test the validity and alignment of the simulated encounters provided by the Patient Simulator to expert human clinical providers. (3) To illustrate the evaluation framework of such an LLM system on the generated realistic, data-driven simulations -- yielding a preliminary assessment of our proposed system. Methods: We first constructed realistic clinical scenarios by deriving patient vignettes from real-world EHR encounters. These vignettes cover a variety of presenting symptoms and underlying conditions. We then evaluate the performance of the Patient Simulator as a simulacrum of a real patient encounter across over 500 different patient vignettes. We leveraged a separate AI agent to provide multi-turn questions to obtain a history of present illness. The resulting multiturn conversations were evaluated by two expert clinicians. Results: Clinicians scored the Patient Simulator as consistent with the patient vignettes in those same 97.7% of cases. The extracted case summary based on the conversation history was 99% relevant. Conclusions: We developed a methodology to incorporate vignettes derived from real healthcare patient data to build a simulation of patient responses to symptom checking agents. The performance and alignment of this Patient Simulator could be used to train and test a multi-turn conversational AI agent at scale.
Problem
Research questions and friction points this paper is trying to address.
Develop a Patient Simulator using real EHR data for AI training
Validate simulated patient encounters against expert clinical assessments
Evaluate conversational AI performance in symptom-checking scenarios
Innovation
Methods, ideas, or system contributions that make the work stand out.
Leveraging real-world EHR data for patient simulation
Multi-turn conversational AI for symptom checking
Clinician-validated synthetic patient encounters
🔎 Similar Papers
No similar papers found.
