Anatomical Region-Guided Contrastive Decoding: A Plug-and-Play Strategy for Mitigating Hallucinations in Medical VLMs

2025-12-18

AI Summary

Medical vision-language models (MedVLMs) are prone to hallucination due to overreliance on textual priors; existing mitigation strategies either require costly expert annotations or apply global, non-targeted interventions, resulting in insufficient clinical robustness. To address this, we propose ARCD—a three-tiered, anatomy-region-mask-guided contrastive decoding framework that performs fine-grained token-, attention-, and logits-level reweighting without model fine-tuning, enabling plug-and-play hallucination suppression. ARCD is the first inference-time intervention framework that jointly achieves anatomical interpretability, training-agnosticism, and cross-modal generalizability. Evaluated on chest X-rays, CT, brain MRI, and ocular ultrasound, ARCD significantly reduces hallucination rates, improves regional understanding accuracy by 12.7%, and boosts diagnostic accuracy by an average of 9.3%.

Technology Category

• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Medical Vision-Language Models (MedVLMs) show immense promise in clinical applicability. However, their reliability is hindered by hallucinations, where models often fail to derive answers from visual evidence, instead relying on learned textual priors. Existing mitigation strategies for MedVLMs have distinct limitations: training-based methods rely on costly expert annotations, limiting scalability, while training-free interventions like contrastive decoding, though data-efficient, apply a global, untargeted correction whose effects in complex real-world clinical settings can be unreliable. To address these challenges, we introduce Anatomical Region-Guided Contrastive Decoding (ARCD), a plug-and-play strategy that mitigates hallucinations by providing targeted, region-specific guidance. Our module leverages an anatomical mask to direct a three-tiered contrastive decoding process. By dynamically re-weighting at the token, attention, and logits levels, it verifiably steers the model's focus onto specified regions, reinforcing anatomical understanding and suppressing factually incorrect outputs. Extensive experiments across diverse datasets, including chest X-ray, CT, brain MRI, and ocular ultrasound, demonstrate our method's effectiveness in improving regional understanding, reducing hallucinations, and enhancing overall diagnostic accuracy.

Problem

• Mitigate hallucinations in medical VLMs
• Provide targeted anatomical region guidance
• Enhance diagnostic accuracy across diverse datasets

Innovation

• Plug-and-play strategy using anatomical region guidance
• Three-tiered contrastive decoding at token, attention, logits
• Dynamic re-weighting to suppress hallucinations in medical VLMs