🤖 AI Summary
This study addresses the scarcity of high-quality speech data that limits the training and evaluation of large language models (LLMs) in real-world medical spoken dialogues. To bridge this gap, the authors introduce MeDial-Speech, a novel dataset comprising over 111 hours of authentic clinician–patient and robot–patient conversational speech across four medical conditions, accompanied by both human and automatic transcripts with word error rates yielding accuracies of 71.1% and 74.7%, respectively. The work further proposes a sentence-selection-based paradigm for evaluating dialogue understanding and benchmarks prominent LLMs—including GPT-5 Mini, DeepSeek-V3, and Claude Sonnet 4—finding that Claude Sonnet 4 achieves the best performance, yet all models exhibit significant overconfidence in their predictions. This resource establishes a foundational benchmark for advancing spoken medical dialogue systems.
📝 Abstract
Large Language Models (LLMs) have brought huge improvements to Artificial Intelligence (AI), which can be applied to general-purpose tasks. However, their application to textual or spoken medical consultations is still an open research problem. This paper proposes MeDial-Speech, a novel speech dataset for training and evaluating Med-AIs that can carry out consultations with patients. It was collected in realistic environments from robot-patient and doctor-patient dialogues, contains 111+ hours of speech data (without data augmentation), and covers four health conditions: Lewy body dementia, heart failure, shoulder pain, and angina. In addition, we propose a dialogue benchmark via sentence selection (with 20 options) to evaluate three state-of-the-art LLMs: GPT-5 mini, DeepSeek-V3, and Claude Sonnet 4. Experimental results reveal that Claude Sonnet 4 is the best in sentence selection, with 71.1% accuracy using manual transcriptions and 74.7% using automatic transcriptions, and that all LLMs are highly overconfident in their probabilistic predictions, regardless of selecting correct or incorrect sentences in medical dialogues. This dataset is free of charge for non-commercial purposes at: https://huggingface.co/datasets/hcuayahu/MeDial-Speech