Problem
Research questions and friction points this paper is trying to address.
medical AI
real-time video consultations
audio-visual interaction
clinical assessment
telehealth
Towards Expert-level Medical AI for Real-time Video Consultations
🤖 AI Summary
This study addresses the limitation of current medical AI systems, which predominantly rely on textual inputs and thus fail to capture critical nonverbal cues present in video-based consultations, thereby constraining their clinical performance. To overcome this, the authors propose AMIE (Video)—a Gemini-based multi-agent system that, for the first time, integrates low-latency audiovisual perception, clinical reasoning, and interactive dialogue within real-world telemedicine video consultations. The work introduces a clinical audiovisual cue evaluation framework tailored for remote care and validates it through a structured Objective Structured Clinical Examination (OSCE). In a randomized controlled trial involving 100 cases, AMIE (Video) demonstrated diagnostic, history-taking, management, and physical observation capabilities comparable to or exceeding those of primary care physicians, with patients expressing greater satisfaction with its explanations and a strong preference for the video-based interaction modality.
📝 Abstract
Audio-visual interaction is the standard for patient-physician consultations, enabling natural communication and effective assessment of illness through non-verbal cues. While text-based AI has shown promise, it discards essential perceptual dimensions and limits patients who cannot articulate symptoms in writing. Early efforts to extend medical AI to audio-visual interaction have demonstrated feasibility but not reached clinician-level performance. Here, we provide the first demonstration of expert-level AI in real-time clinical video consultations using AMIE (Articulate Medical Intelligence Explorer) in a video configuration. AMIE (Video) is a Gemini-based multi-agent system integrating low-latency dialogue, clinical reasoning, and real-time audio-visual perception. To guide development, we established a taxonomy and automated evaluations for clinical audio-visual cues in telehealth settings. In a randomized Objective Structured Clinical Examination (OSCE) study with 30 primary care physicians (PCPs), 15 patient actors and 100 clinical scenarios, we compared AMIE (Video), its text-only counterpart AMIE (Text), and PCPs consulting via video. Clinical evaluators rated AMIE (Video) on par or better than PCPs in history-taking, diagnosis, management, and physical observation and examination. Patient actors preferred AMIE's approach to assessing and explaining conditions, while PCPs were preferred for rapport and partnership building. In modality ablation, patient actors preferred AMIE (Video)'s interface over text chat for communicative effectiveness, convenience, and feeling understood. Limitations remain in fine anatomical precision, subtle affective nuances, and high-frequency movements. While further research is needed before real-world translation, these results mark an important milestone toward AI systems capable of augmenting care across the sensory complexity of clinical practice.
Innovation
Methods, ideas, or system contributions that make the work stand out.
audio-visual medical AI
real-time video consultation
multi-agent clinical reasoning
automated clinical cue evaluation
expert-level diagnostic performance
🔎 Similar Papers
No similar papers found.
