Dialogue is Better Than Monologue: Instructing Medical LLMs via Strategical Conversations

2025-01-29

AI Summary

Current medical AI systems exhibit insufficient reasoning robustness in complex, noisy real-world clinical settings, primarily due to overreliance on static text-based question answering and the absence of evidence-based, iterative diagnostic reasoning. Method: We introduce the first noise-robust, multi-turn dialogue benchmark tailored for authentic clinical diagnosis. It innovatively reformulates the USMLE question bank into strategic, clinician-like dialogues and incorporates clinical-logic-driven noise injection and difficulty stratification. We further propose a dialogue-oriented fine-tuning paradigm to systematically enhance model resilience. Contribution/Results: Empirical evaluation demonstrates that our approach improves accuracy by 9.64% on multi-turn reasoning tasks and by 6.18% under noisy conditions, while significantly boosting clinical consistency and cross-scenario generalization—establishing the first systematic validation of dialogue-based fine-tuning for improving robustness and diagnostic fidelity in medical large language models.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Current medical AI systems often fail to replicate real-world clinical reasoning, as they are predominantly trained and evaluated on static text and question-answer tasks. These tuning methods and benchmarks overlook critical aspects like evidence-based reasoning and handling distracting information. To bridge this gap, we introduce a novel benchmark that simulates real-world diagnostic scenarios, integrating noise and difficulty levels aligned with USMLE standards. Moreover, we explore dialogue-based fine-tuning, which transforms static datasets into conversational formats to better capture iterative reasoning processes. Experiments show that dialogue-tuned models outperform traditional methods, with improvements of $9.64%$ in multi-round reasoning scenarios and $6.18%$ in accuracy in a noisy environment. Our findings highlight dialogue tuning as a promising approach for advancing clinically aligned and robust medical AI systems.

Problem

• Medical AI
• Complex Scenario Understanding
• Evidence-based Decision Making

Innovation

• Dialogue-based Learning
• Medical AI
• Interactive Training Method