Unified Multimodal Model for Brain MRI Imputation and Understanding

2026-06-15

AI Summary

This work addresses the challenges of scarce high-quality multimodal training data and frequent clinical modality missingness in brain MRI analysis by proposing UniBrain, a unified multimodal large model. The approach jointly optimizes modality completion and semantic understanding through an interleaved description-augmented data stream for autoregressive training. It introduces a novel self-alignment learning strategy that operates without fine-grained annotations and a dynamic hidden state mechanism to mitigate exposure bias in long-context modeling. Extensive experiments across multiple brain MRI datasets demonstrate that UniBrain consistently achieves superior performance in image imputation, semantic interpretation, and disease diagnosis under various modality-missing scenarios, significantly outperforming existing methods.

Abstract

Multimodal large language models (MLLMs) hold great potential for medicine, as they inherit knowledge from LLM and allow multiple data modalities to be integrated, analysed and interpreted in natural language. However, the field of medical MLLMs is constrained by non-trivial challenges, notably the scarcity of high-quality training data and the frequent occurrence of missing data in the real-world clinical setting. Here, we propose a novel unified multimodal model, UniBrain, for brain magnetic resonance image (MRI) analysis. To address potential missing brain MRI modalities, we employ a unified training strategy to perform joint imaging modality imputation and brain image understanding. During training, an interleaved and description-enriched data flow is constructed to train the model in an autoregressive manner, enabling medical reasoning with generated multimodal data. A self-alignment strategy is introduced to leverage dense image embeddings to learn fine-grained anatomical features without requiring detailed image captions. Furthermore, we propose a dynamic hidden state mechanism to alleviate the exposure bias during long-context multimodal inference. Extensive experiments on multi-disease brain MRI dataset demonstrate that UniBrain achieves high performance for brain image imputation, understanding, and disease diagnosis under various extents of modality incompleteness.

Problem

• multimodal large language models
• brain MRI
• missing data
• medical imaging
• data scarcity

Innovation

• multimodal imputation
• unified training
• self-alignment
• dynamic hidden state
• medical MLLM