Are LLMs Ready to Assist Physicians? PhysAssistBench for Interactive Doctor-Patient-EHR Assistance

2026-06-16

AI Summary

Current evaluations of medical large language models are often confined to isolated capabilities, lacking comprehensive assessment in scenarios involving collaborative interactions among physicians, patients, and electronic health records (EHRs). This work proposes PhysAssistBench, the first benchmark framework tailored to this triadic interaction setting. Built upon real-world MIMIC-IV clinical notes, it introduces an agent-based patient simulator and a multi-turn dialogue generation pipeline, enhanced with clinical factuality preservation mechanisms and physician review protocols to produce 1,296 high-quality bilingual interaction samples. Experimental results reveal that state-of-the-art models exhibit substantial performance gaps in such integrated assistance tasks, highlighting critical bottlenecks in their ability to synergistically apply medical knowledge, comprehend communicative intent, and effectively invoke external tools.

Abstract

The most plausible near-term role of medical LLMs is to assist rather than replace physicians, yet current evaluations often test isolated capabilities: clinical knowledge, EHR system interaction, or patient communication. Physician assistance instead requires coordinating these capabilities within the same interaction, where physicians issue underspecified requests, patients describe symptoms ambiguously, and EHR systems demand precise tool use. We introduce PhysAssistBench, a benchmark for interactive doctor-patient-EHR assistance. Built from real MIMIC-IV cases, PhysAssistBench uses a scalable pipeline to construct agentic patients: interactive, record-grounded agents that turn static EHR records into multi-turn clinical scenarios while preserving clinical factuality. PhysAssistBench provides a curated bilingual evaluation set of 1,296 manually reviewed and physician-validated turns. Experiments with leading LLMs show that current models remain unreliable in this setting, which exposes a key bottleneck for clinical LLMs: reliable assistance requires coordination across knowledge, communication, and systems, not isolated gains in any of them.

Problem

• medical LLMs
• physician assistance
• interactive clinical scenarios
• EHR interaction
• multimodal coordination

Innovation

• PhysAssistBench
• interactive clinical assistance
• agentic patients
• EHR integration
• multimodal coordination