MedDialogRubrics: A Comprehensive Benchmark and Evaluation Framework for Multi-turn Medical Consultations in Large Language Models

2026-01-06

arXiv.org

AI Summary

Existing evaluation frameworks struggle to effectively assess large language models’ (LLMs’) capabilities in information gathering and diagnostic reasoning during multi-turn medical consultations. To address this gap, this work introduces a benchmark comprising 5,200 synthetic clinical cases and over 60,000 fine-grained scoring criteria, leveraging a multi-agent system to simulate authentic patient dialogues and integrating evidence-based medical guidelines to define “mandatory inquiry” dimensions. The study innovatively incorporates a clinician-validated LLM scoring pipeline, a dynamic hallucination detection and correction mechanism, and a high-quality dialogue generation strategy based on rejection sampling. Comprehensive evaluations of leading LLMs reveal significant shortcomings in current systems during multi-turn medical interactions, underscoring the urgent need for architectural improvements in dialogue management rather than reliance solely on base model fine-tuning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Medical conversational AI (AI) plays a pivotal role in the development of safer and more effective medical dialogue systems. However, existing benchmarks and evaluation frameworks for assessing the information-gathering and diagnostic reasoning abilities of medical large language models (LLMs) have not been rigorously evaluated. To address these gaps, we present MedDialogRubrics, a novel benchmark comprising 5,200 synthetically constructed patient cases and over 60,000 fine-grained evaluation rubrics generated by LLMs and subsequently refined by clinical experts, specifically designed to assess the multi-turn diagnostic capabilities of LLM. Our framework employs a multi-agent system to synthesize realistic patient records and chief complaints from underlying disease knowledge without accessing real-world electronic health records, thereby mitigating privacy and data-governance concerns. We design a robust Patient Agent that is limited to a set of atomic medical facts and augmented with a dynamic guidance mechanism that continuously detects and corrects hallucinations throughout the dialogue, ensuring internal coherence and clinical plausibility of the simulated cases. Furthermore, we propose a structured LLM-based and expert-annotated rubric-generation pipeline that retrieves Evidence-Based Medicine (EBM) guidelines and utilizes the reject sampling to derive a prioritized set of rubric items ("must-ask"items) for each case. We perform a comprehensive evaluation of state-of-the-art models and demonstrate that, across multiple assessment dimensions, current models face substantial challenges. Our results indicate that improving medical dialogue will require advances in dialogue management architectures, not just incremental tuning of the base-model.

Problem

• medical dialogue systems
• large language models
• diagnostic reasoning
• evaluation benchmark
• multi-turn consultation

Innovation

• multi-agent simulation
• hallucination correction
• evidence-based rubrics
• privacy-preserving benchmark
• dialogue management architecture