AIPatient Arena: EHR-grounded evaluation of large language models in end-to-end clinical consultation workflows

2026-06-15

AI Summary

Current evaluations of medical large language models are largely confined to static, single-turn settings or focus solely on final answer accuracy, failing to capture the complexity of real-world clinical practice—characterized by multi-turn interactions, uncertain information, and dynamic reasoning. This work introduces a novel end-to-end evaluation framework that integrates electronic health records (EHRs) into multi-turn clinical dialogues, leveraging patient knowledge graphs to establish a structured assessment protocol spanning eight clinical competency dimensions. The framework combines simulated multi-turn conversations, expert human ratings, and cross-distribution validation cohorts. Experiments across a primary cohort of 437 cases and two external cohorts reveal that while models perform adequately in history-taking and ethical compliance, they exhibit significant deficiencies in handling ambiguous responses, ensuring comprehensive information coverage, and executing diagnostic reasoning—highlighting that final-answer accuracy alone is insufficient for assessing clinical utility.

Abstract

Large language models (LLMs) are increasingly considered for use in clinical consultation tasks, yet most medical evaluations remain static, single-turn, or narrowly outcome-based, limiting their ability to reflect the sequential, uncertain, and interactive nature of real-world care. Here, we propose AIPatient Arena, an EHRs-grounded evaluation framework for assessing the clinical utility of LLMs across eight dimensions of clinical competence. The framework integrates EHR data into patient-specific knowledge graphs, enabling multi-turn physician-patient interactions. We applied AIPatient Arena on a primary cohort of 437 patients and two out-of-distribution validation cohorts of 119 and 67 patients. We observe that LLMs performed well in medical interview questioning skills (QS; mean scores, 4.43-4.99/5), ethical and professional conduct (ET; 4.38-4.93/5), and clarity and transparency of clinical explanations (EX; 3.80-4.72/5). Performance was moderate in information integration (II; 3.19-4.21/5) and medication safety and justification (MS; 3.13-3.78/5), but persistent weaknesses were observed in handling of ambiguous patient responses (HR; 2.57-3.32/5), information coverage (IC; 2.08-3.02/5), and diagnostic accuracy and reasoning (Dx; 2.63-3.55/5). Process-based evaluation revealed recurrent interaction failures, including repetitive questioning, omission of past medical history, and inadequate handling of uncertainty. Richer conversational context improved diagnostic reasoning but yielded limited gains in treatment planning. These findings indicate that final-answer accuracy alone is insufficient for evaluating clinical readiness and highlight the importance of assessing how models gather, interpret, and communicate information throughout a consultation. AIPatient Arena provides an EHR-grounded framework for workflow-oriented pre-deployment evaluation of medical LLMs.

Problem

• clinical consultation
• large language models
• evaluation framework
• electronic health records
• diagnostic reasoning

Innovation

• EHR-grounded evaluation
• clinical consultation workflow
• multi-turn interaction
• knowledge graph
• process-based assessment