Vision-language models for chest radiography do not always need the image

2026-06-16

AI Summary

Although existing medical vision-language models achieve high accuracy in chest X-ray diagnosis, they may rely primarily on textual priors rather than genuinely leveraging image information. To address this, this work proposes the first causal auditing framework for medical multimodal models, systematically evaluating their dependence on visual input through image interventions—such as occluding critical regions or swapping images with others sharing the same label—combined with three behavioral metrics. Experiments across text-only and multimodal models ranging from 7B to 119B parameters reveal that most systems fail to effectively “ground” their predictions in the image: text-only models perform nearly as well as the best multimodal counterparts, only 5 out of 9 models selectively utilize visual information at a level comparable to radiologists, and model confidence reliably flags ungrounded answers only when the model actually uses the image. The study argues that clinical deployment should prioritize grounding over mere accuracy as the key criterion for model adoption.

Abstract

Medical vision-language models report strong chest radiograph accuracy, and this is increasingly read as evidence that they use the image. That inference is unsafe: a model exploiting finding-name priors scores like one that reads the scan, and no standard benchmark separates them. We introduce a causal audit that intervenes on the image, occluding the relevant region, occluding an irrelevant one, and swapping in another patient's same-label scan, and combines three behavioral metrics to test whether a correct answer depends on the image. Across nine systems, a text-only model with no image access reaches within 5.7 accuracy points of the best multimodal one, and a 119-billion-parameter multimodal model is statistically indistinguishable from a 7-billion text-only baseline. The audit splits the cohort into three models that ignore the image, one that is unstable, and five that use it selectively, for a subset of findings; the categories hold across a second dataset, resolution, and prompt phrasing. Against board-certified radiologists, a text-only model is statistically indistinguishable from a radiologist's accuracy while grounding at zero, whereas the image-using models ground at radiologist-comparable rates. Reported confidence flags ungrounded answers only when a model uses the image. Grounding audits, not accuracy, should gate clinical deployment.

Problem

• vision-language models
• chest radiography
• image grounding
• finding-name priors
• model auditing

Innovation

• causal audit
• vision-language models
• chest radiography
• grounding
• finding-name priors