Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs
This work addresses the challenge that existing medical vision-language models (VLMs) often fail to distinguish between reliance on genuine visual evidence and exploitation of textual shortcuts in visual question answering (VQA). To disentangle visual and textual contributions, the authors propose a counterfactual evaluation framework that replaces original images with controlled substitutes—such as blank images, pixel-shuffled variants, or hard negatives retrieved via CLIP—and introduce a contrastive grounding objective (CGO) training strategy to enhance visual dependency. They further define novel metrics, including Visual Reliance Score (VRS) and Visual Hallucination Rate (VHR), and optimize the Qwen2.5-VL-7B model using LoRA fine-tuning with CLIP-based hard negatives. Experiments demonstrate that the resulting CORAL model achieves a 6.7% absolute improvement in macro accuracy and an 8.0% reduction in hallucination rate across four medical VQA benchmarks, significantly outperforming baselines with stronger generalization capability.