CLExEval: A Human-in-the-Loop Framework for Qualitative Evaluation of LLM Clinical Reasoning

2026-06-30

AI Summary

This work addresses the vulnerability of large language models (LLMs) to “evaluation hallucination” in clinical reasoning—where fluent yet erroneous explanations mask diagnostic inaccuracies. To this end, the authors propose CLExEval, a novel framework that integrates 5,600 physician annotations and 200 reasoning trajectories to qualitatively analyze LLM reasoning in rare disease diagnosis. Through progressive information masking, human-in-the-loop evaluation, and LLM-as-a-Judge validation, the study identifies three failure modes: verbosity bias under information scarcity, a hidden knowledge paradox stemming from failed expert knowledge retrieval, and misalignment between internal reasoning and final output. Experiments reveal that GPT-4o-mini’s accuracy drops sharply from 95.0% to 32.5% under limited information, and 68.6% of correct reasoning chains fail to translate into accurate final answers, highlighting a significant overestimation of clinical reliability by automated evaluation metrics.

Abstract

Large Language Models (LLMs) achieve strong results on many medical benchmarks, but their clinical reasoning remains difficult to evaluate reliably. A central risk is an evaluation illusion: fluent and well-structured explanations can appear clinically convincing even when the final diagnosis is incorrect. We introduce CLExEval, a human-in-the-loop framework for evaluating LLM clinical reasoning under progressive information masking. CLExEval combines 5,600 expert-physician annotations with 200 clinical reasoning traces derived from 40 rare diagnostic cases. Our analysis identifies three recurring failure patterns: (i) verbosity bias, where GPT-4o-mini's diagnostic accuracy drops from 95.0% to 32.5% under information scarcity; (ii) a hidden knowledge paradox, where a specialist model reaches 92.5% maximum diagnostic potential but fails to retrieve that knowledge reliably in verbose contexts; and (iii) a 68.6% reasoning-to-output mismatch, where correct diagnoses appear in reasoning traces but are not reflected in final answers. We further evaluate the LLM-as-a-Judge paradigm on a human-verified failure set (n = 142). GPT-4o-mini approved 47.9% of clinically incorrect outputs, while HuatuoGPT-o1 approved all validly scored failures and showed a positive self-preference bias. These results suggest that standalone automated clinical evaluations can substantially overestimate clinical reliability without expert-grounded validation.

Problem

• clinical reasoning
• evaluation illusion
• large language models
• human-in-the-loop
• diagnostic accuracy

Innovation

• human-in-the-loop evaluation
• clinical reasoning
• information masking
• LLM-as-a-Judge
• reasoning-to-output mismatch