Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs

2026-07-03

AI Summary

This work addresses the challenge that existing medical vision-language models (VLMs) often fail to distinguish between reliance on genuine visual evidence and exploitation of textual shortcuts in visual question answering (VQA). To disentangle visual and textual contributions, the authors propose a counterfactual evaluation framework that replaces original images with controlled substitutes—such as blank images, pixel-shuffled variants, or hard negatives retrieved via CLIP—and introduce a contrastive grounding objective (CGO) training strategy to enhance visual dependency. They further define novel metrics, including Visual Reliance Score (VRS) and Visual Hallucination Rate (VHR), and optimize the Qwen2.5-VL-7B model using LoRA fine-tuning with CLIP-based hard negatives. Experiments demonstrate that the resulting CORAL model achieves a 6.7% absolute improvement in macro accuracy and an 8.0% reduction in hallucination rate across four medical VQA benchmarks, significantly outperforming baselines with stronger generalization capability.

Abstract

Large vision language models (VLMs) report strong accuracy on medical question-answering, yet it remains unclear whether they reason from visual evidence or exploit textual shortcuts. We introduce a counterfactual evaluation framework that decouples visual and textual contributions by substituting input images with controlled surrogates blank, pixel-shuffled, image-absent, and CLIP-retrieved hard negatives and derive a suite of grounding metrics including the Visual Reliance Score (VRS) and Visual Hallucination Rate (VHR). We further introduce CORAL (COntrastive Retrieval-Augmented Learning), a 7B-parameter LoRA fine-tune of Qwen2.5-VL-7B trained with a Contrastive Grounding Objective (CGO) that penalises answer invariance under hard-negative image swaps. On a paired controlled evaluation across four closed-form medical VQA benchmarks (PathVQA, PMC-VQA, SLAKE, VQA-RAD; n=400 total), CORAL improves macro accuracy by +6.7 pp (P(Delta>0)=0.988) and reduces VHR by 8.0 pp (P<0.001) over the matched Qwen2.5-VL-7B base; neither MedVLThinker RL variant achieves a significant gain on either metric. Cross-domain diagnostics further reveal that image substitution costs only <=6.5 pp on medical benchmarks versus 48-61 pp on general-domain tasks, situating the grounding gap that CGO targets. We discuss evaluation limitations openly including train/eval benchmark overlap and underpowered secondary metrics and release our framework, training code, and model weights to support reproducible grounding audits of medical VLMs.

Problem

• Medical Vision Language Models
• Visual Grounding
• Textual Shortcuts
• Visual Reliance
• Hallucination

Innovation

• Counterfactual Evaluation
• Visual Grounding
• Hard-Negative Contrastive Training
• Medical Vision Language Models
• Visual Hallucination