Resolution Meets Reduction: Efficient Visual Context for 3D Radiology Report Generation

2026-08-09

AI Summary

This study addresses the challenges of redundant visual tokens and low computational efficiency in 3D CT report generation due to processing full-volume inputs. The authors systematically evaluate combinations of 3D/2D vision encoders (CNNs and ViTs), token compression projectors (up to 64× compression), and language models ranging from 1.7B to 4B parameters under a fixed token budget. They propose an anatomy-guided region-of-interest cropping and resolution adaptation strategy that substantially improves visual context representation. Experiments demonstrate that this cropping approach yields the most stable performance, and that high-resolution inputs synergize effectively with advanced projectors such as PerceiverResampler. The method achieves new state-of-the-art clinical macro F1 scores of 49.5 on CT-RATE and 49.0 on Merlin, with an average improvement of 3.7 points across 19 out of 20 evaluated settings.

Abstract

Vision-language models offer a promising path toward automating radiology report generation, but applying them to full 3D CT volumes poses substantial computational challenges. Modern foundation vision encoders (VEs) can produce tens of thousands of vision tokens per scan, making the visual sequence passed to the large language model (LLM) a primary computational bottleneck. Vision-to-language projectors can compress this sequence to reduce computation, but may discard clinically relevant detail; conversely, effective compression can accommodate higher-resolution inputs while keeping the downstream token count fixed. How this vision-token budget should be allocated across input field of view, spatial resolution, and vision-to-language projection therefore remains an open design question. We systematically evaluate four heterogeneous VEs (CNN- and ViT-based), five token-reducing projectors at up to 64x compression alongside a non-reducing MLP projector baseline, and five instruction-tuned LLMs (1.7B--4B) on two large-scale CT report datasets (CT-RATE and Merlin). At matched LLM token budgets, anatomy-guided region of interest cropping is the most consistent strategy, improving clinical macro F1 in 19 of 20 settings by +3.7 points on average for the 3D ViT Primus encoder and +1.1 for the slice-based 2D ViT Curia encoder. Increasing input resolution further is strongly projector-dependent: the PerceiverResampler, paired with higher-resolution Curia features, yields the strongest configuration in the resolution study on both datasets. Our best configurations achieve state-of-the-art clinical macro F1 on the test sets, reaching 49.5 on CT-RATE and 49.0 on Merlin. Code and models will be published upon publication.

Problem

• 3D radiology report generation
• vision-language models
• token compression
• visual context efficiency
• CT volume analysis

Innovation

• vision-language models
• token compression
• region of interest cropping
• 3D radiology report generation
• PerceiverResampler