MedBench v5: A Dynamic, Process-Oriented, and Hallucination-Aware Benchmark for Clinical Multimodal Models

2026-06-23

AI Summary

Current evaluations of medical AI systems lack transparency in reasoning processes, fine-grained assessment of atomic clinical skills, and robust hallucination detection. This work proposes the first dynamic, process-oriented multimodal evaluation framework for clinical AI, integrating language, vision-language, and agent-based systems. It introduces a dual-dimensional assessment combining clinical cognitive responses and medical atomic skills, a five-node dynamic reasoning audit trail, three switchable information-flow perturbation strategies, and a cross-stage hallucination propagation tracking mechanism to jointly analyze model reasoning stability and hallucination trajectories. Experimental results reveal that high overall performance does not necessarily imply robust reasoning: information-flow stress significantly impairs contradiction detection, diagnostic updating, and self-correction of hallucinations, suggesting that seemingly stable final outputs may merely reflect superficial consistency rather than genuine reasoning reliability.

Abstract

Existing medical AI benchmarks lack process visibility, atomic skill evaluation, and integrated hallucination detection. We introduce MedBench v5, a redesigned benchmark for clinical multimodal models (language, vision-language, and agent systems) that moves from static QA to dynamic, process-oriented evaluation. MedBench v5 features: (1) a dual-dimensional framework combining Clinical Cognitive Responsiveness (14 sub-dimensions) and Medical Atomic Skills (4 agent environments), covering 63 tasks; (2) three switchable information-flow stressors (omission, contradiction, evidence delay) for factorized degradation analysis; (3) a dynamic process audit protocol with five reasoning nodes that produces model-specific failure fingerprints; (4) hallucination propagation monitoring across initiation, propagation, anchoring, and contradiction interaction-capturing silent hallucination. Experiments on frontier models show that strong overall task performance does not guarantee process stability: stressors mainly disrupt contradiction detection, diagnosis updating, hallucination propagation, and contradiction-based self-correction, while final evidence grounding can remain superficially stable. MedBench v5 provides a unified infrastructure for capability profiling, controllable stress testing, process auditing, and hallucination trajectory analysis in clinical AI evaluation.

Problem

• medical AI benchmark
• process-oriented evaluation
• hallucination detection
• multimodal models
• clinical reasoning

Innovation

• dynamic evaluation
• process-oriented benchmarking
• hallucination propagation
• clinical multimodal models
• information-flow stressors