Argus: Benchmarking and Enhancing Vision-Language Models for 3D Radiology Report Generation

2024-06-11

AI Summary

Automated radiology report generation from 3D CT volumes lacks large-scale public benchmarks and systematic vision-language model (VLM) training strategies. Method: We introduce CT-3DRRG, the first publicly available dataset of paired 3D CT volumes and clinical reports, and propose a VLM training paradigm tailored to high-resolution 3D medical imaging—systematically evaluating visual encoders (3D CNNs vs. ViTs), visual token compression techniques (pooling vs. PCA), and multi-scale progressive training. We further release the efficient Argus model family, supporting inputs up to 512×512×256 voxels. Results: On CT-3DRRG, Argus substantially outperforms prior methods (BLEU-4 +12.3%, CIDEr +18.7%), establishing for the first time the critical impact of visual encoder architecture and token compression on clinical report fidelity. This work provides a reproducible benchmark and empirically grounded training guidelines for 3D medical VLMs.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Argumentation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Automatic radiology report generation holds significant potential to streamline the labor-intensive process of report writing by radiologists, particularly for 3D radiographs such as CT scans. While CT scans are critical for clinical diagnostics, they remain less explored compared to 2D radiographs. To date, there has been no comprehensive benchmark for 3D radiograph report generation (3DRRG), nor sufficient investigation into the optimal training strategies for Vision Language Models (VLMs) in this context, particularly with respect to vision encoder choices, visual token compression, and model scaling. In this work, we make three key contributions. We curate **CT-3DRRG**, the largest **publicly** available 3D CT-report dataset, establishing a robust and diverse benchmark for evaluating VLM performance on 3DRRG. Furthermore, we propose a comprehensive training recipe for building high-performing VLMs for 3DRRG, exploring key factors such as vision encoder pretraining strategies, visual token compression, and the impact of data&model scale. Guided by these findings, we introduce **Argus**, a state-of-the-art family of VLMs that achieve superior performance across different model sizes and input 3D medical image resolutions, efficiently processing high-resolution 3D images up to $512 imes 512 imes 256$[^1].

Problem

• Benchmarking 3D radiology report generation
• Optimizing Vision-Language Models for 3D CT scans
• Developing efficient training strategies for 3DRRG

Innovation

• curated CT-3DRRG dataset
• comprehensive VLM training recipe
• introduced Argus VLMs