Evaluating Large Language Models on Misconceptions in Multi-Turn Medical Conversations

2026-07-14

AI Summary

Existing evaluation frameworks struggle to effectively assess large language models’ ability to identify and correct patients’ misconceptions in multi-turn physician–patient dialogues—a capability critical for safe medical communication. This work introduces the first evaluation paradigm specifically designed for misconception correction in multi-turn clinical conversations, presenting ThReadMed-QA, a dataset comprising 2,437 dialogue threads. The authors employ a hybrid evaluation strategy combining rule-based LLM-as-a-Judge scoring with oracle-based analysis. Experimental results reveal that state-of-the-art models, such as GPT-5 and Claude-Haiku, achieve an initial correction rate of 85% in the first turn, yet this performance sharply declines to approximately 50% after two subsequent turns, exposing significant degradation and error propagation risks in extended dialogues.

Abstract

Patients seeking medical information often ask questions that embed incorrect assumptions or misconceptions. In such cases, safe medical communication requires not only answering the question, but identifying and correcting the underlying false belief. These interactions naturally unfold over multiple turns, a pattern now mirrored in interactions with LLMs. Yet current evaluation frameworks do not capture model behavior in these settings, where misconceptions can emerge, persist, or evolve over the course of a conversation. Whether LLMs can reliably correct such misconceptions over time remains largely unexamined. To study this, we introduce ThReadMed-QA, a multi-turn medical dialogue dataset of 2,437 patient-physician conversation threads comprising 8,204 question-answer pairs, derived from real patient interactions on AskDocs. This dataset enables systematic evaluation of whether models can detect and correct misconceptions under a multi-turn context. We evaluate five LLMs using a rubric-based LLM-as-a-Judge framework that scores responses based on their ability to identify and correct misconceptions. Our experiments reveal a consistent pattern: even frontier models that can address misconceptions in a single interaction degrade substantially over subsequent turns. GPT-5 and Claude-Haiku correct these false presuppositions around 85% on initial questions but drop to roughly 50% within two follow-ups. An oracle analysis replacing prior model outputs with physician responses shows that much of the degradation is driven by error propagation, while performance remains imperfect even under correct context. Even when models tend to correct misconceptions initially, their performance degrades substantially over later turns, leading to inconsistent and potentially unsafe guidance in patient-facing settings and highlighting the need for evaluation frameworks that capture multi-turn behavior.

Problem

• misconceptions
• multi-turn conversations
• large language models
• medical dialogue
• evaluation framework

Innovation

• multi-turn dialogue
• misconception correction
• medical LLM evaluation
• error propagation
• ThReadMed-QA