SHOVIR: A Benchmark for Evaluating Vision Shortcut Learning in Radiology Report Generation

2026-06-29

AI Summary

This study addresses the limitations of current radiology report generation models, which rely on holistic evaluation metrics that fail to verify whether diagnoses are grounded in genuine pathological visual evidence, rendering them susceptible to spurious correlations or prior biases. To tackle this issue, the authors introduce the SHOVIR benchmark, incorporating region-level CheXpert labels on MIMIC-CXR and PadChest-GR datasets, and design image- and disease-level occlusion experiments to systematically distinguish between “direct shortcuts” and “contextual shortcuts”—two failure modes of visual dependency. Through a spatially aligned evaluation framework, the work exposes a critical blind spot in existing methods: their neglect of fine-grained regional perception. Experiments across eight state-of-the-art vision-language models reveal that fluent report generation does not necessarily indicate reliable visual grounding, as top-performing models may still exhibit shallow reliance on image evidence.

Abstract

Current evaluation protocols for Vision-Language Models (VLMs) in Radiology Report Generation (RRG) rely on report-level metrics that measure lexical overlap or aggregate clinical correctness. However, such metrics do not test whether individual diagnostic statements stem from the actual pathological evidence visible in the image. This allows models to achieve competitive scores by exploiting learned priors or spurious correlations, a failure mode we refer to as vision shortcut. We introduce SHOVIR, a benchmark for evaluating vision shortcut behavior in RRG. SHOVIR extends two spatially annotated chest X-ray datasets, MIMIC-CXR and PadChest-GR, with per-box CheXpert labels, and defines image-level and disease-level occlusion experiments that contrast baseline performance on clean images against localized, region-specific perturbations. Comparing predictions across these conditions isolates two failure modes at the disease-class level: direct shortcuts, where a finding persists after its visual evidence is removed, and contextual shortcuts, where detection degrades once co-occurring pathologies are occluded despite the target region remaining intact. Benchmarking eight state-of-the-art VLMs, we find that shortcut behavior varies substantially across architectures and datasets. Models achieving the highest baseline report quality do not necessarily rank highest in spatial grounding, revealing that clinically fluent generation can coexist with shallow reliance on visual evidence. These findings expose a blind spot in current RRG evaluation and motivate region-aware assessment protocols.

Problem

• vision shortcut
• radiology report generation
• visual grounding
• evaluation benchmark
• spurious correlation

Innovation

• vision shortcut learning
• radiology report generation
• spatial grounding
• occlusion benchmark
• vision-language models