U-VLM: Hierarchical Vision Language Modeling for Report Generation

2026-02-28

AI Summary

This work addresses the challenge of effectively integrating multi-scale visual information in 3D medical image report generation with current vision-language models, which often underutilize the potential of segmentation-pretrained encoders. The authors propose U-VLM, a novel framework that, for the first time, combines segmentation pretraining with a staged progressive training strategy—sequentially advancing from segmentation to classification and finally to report generation. A multi-layer visual feature injection mechanism is introduced to align and inject hierarchical features from a U-Net encoder into corresponding layers of a lightweight 0.1B-parameter language decoder, enabling hierarchical vision-language modeling. Without requiring unified annotations, U-VLM leverages heterogeneous medical data effectively, achieving state-of-the-art performance on CT-RATE and AbdomenAtlas 3.0, with significant improvements in F1 and BLEU scores over existing baselines, thereby demonstrating that thoughtfully designed visual pretraining can surpass approaches relying solely on large language models.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Automated radiology report generation is key for reducing radiologist workload and improving diagnostic consistency, yet generating accurate reports for 3D medical imaging remains challenging. Existing vision-language models face two limitations: they do not leverage segmentation-pretrained encoders, and they inject visual features only at the input layer of language models, losing multi-scale information. We propose U-VLM, which enables hierarchical vision-language modeling in both training and architecture: (1) progressive training from segmentation to classification to report generation, and (2) multi-layer visual injection that routes U-Net encoder features to corresponding language model layers. Each training stage can leverage different datasets without unified annotations. U-VLM achieves state-of-the-art performance on CT-RATE (F1: 0.414 vs 0.258, BLEU-mean: 0.349 vs 0.305) and AbdomenAtlas 3.0 (F1: 0.624 vs 0.518 for segmentation-based detection) using only a 0.1B decoder trained from scratch, demonstrating that well-designed vision encoder pretraining outweighs the benefits of 7B+ pre-trained language models. Ablation studies show that progressive pretraining significantly improves F1, while multi-layer injection improves BLEU-mean. Code is available at https://github.com/yinghemedical/U-VLM.

Problem

• radiology report generation
• 3D medical imaging
• vision-language modeling
• multi-scale visual features
• segmentation-pretrained encoders

Innovation

• hierarchical vision-language modeling
• progressive pretraining
• multi-layer visual injection
• U-Net encoder
• radiology report generation