Revisiting LLM Adaptation for 3D CT Report Generation: A Study of Scaling and Diagnostic Priors

2026-06-15

AI Summary

Directly fine-tuning large language models (LLMs) for 3D medical imaging report generation is prone to overfitting and clinical hallucinations, and struggles to bridge the gap between volumetric data complexity and clinical semantics. To address these challenges, this work proposes RAD3D-Prefix, a framework that freezes the LLM while introducing a lightweight diagnostic prior conditioning module to fuse 3D image embeddings with multi-label diagnostic classification logits for efficient prefix modulation. This approach substantially reduces trainable parameters, enhances out-of-domain generalization, and outperforms existing parameter-efficient fine-tuning methods in both automated metrics and clinical expert evaluations, thereby demonstrating the superiority of the “frozen large model + lightweight adapter” paradigm for 3D medical report generation.

Abstract

Recent advances in multimodal learning, including large language models (LLMs) and vision-language models (VLMs), have demonstrated strong adaptability to natural images. However, extending their use to the medical domain, particularly for volumetric (3D) images, is challenging due to high computational complexity, volumetric dependencies and the semantic gap between visual features and clinical terminology. Naively fine-tuning LLMs on limited medical data often leads to overfitting and clinical hallucination, where linguistic fluency is prioritized over clinical factuality. In this study, we investigate parameter-efficient adaptation strategies for volumetric CT report generation and introduce RAD3D-Prefix, a lightweight diagnostic-prior conditioning framework that minimizes the need for extensive parameter training. This module integrates image embeddings with multi-label diagnostic classification logits, preserving critical clinical details while bridging the semantic gap. By keeping the LLM frozen, our method requires minimal trainable parameters and mitigates the risk of overfitting on small, domain-specific datasets. Through a systematic study spanning LLMs from 96.1M to 1.6B parameters, we find that fine-tuning is most beneficial for smaller LLMs, whereas freezing larger (~1B+ LLMs and training only lightweight projection layers provides a superior trade-off between performance, generalization, and computational efficiency. Across multiple automatic metrics and a clinical reader study, RAD3D-Prefix outperforms comparable parameter-efficient baselines and demonstrates strong out-of-domain generalization while using substantially fewer trainable parameters than fully fine-tuned alternatives.

Problem

• 3D CT report generation
• large language models
• medical domain adaptation
• clinical hallucination
• semantic gap

Innovation

• parameter-efficient adaptation
• diagnostic priors
• 3D CT report generation
• frozen LLM
• semantic gap bridging