Modeling Local, Global, and Cross-Modal Context in Multimodal 3D MRI

2026-06-25

AI Summary

This work addresses the challenge of effectively fusing complementary information from multimodal 3D brain MRI data, which exhibit limited sample sizes and high anatomical and pathological diversity. To this end, the authors propose MICViT, a novel model that, for the first time in 3D multimodal MRI analysis, jointly models intra- and inter-modal as well as local and global contextual relationships. MICViT achieves efficient multimodal fusion through four distinct attention mechanisms: modality-specific local and global attention, and cross-modal local and global attention. Built upon a 3D Vision Transformer architecture, MICViT significantly outperforms both CNN- and Transformer-based baselines on three large-scale brain age prediction datasets, including UK Biobank, with performance consistently improving as the number of input modalities increases.

Abstract

Brain MRI poses a fundamental challenge for machine learning: models must learn from high-dimensional 3D data spanning multiple co-registered modalities, despite the limited sample sizes typical of neuroimaging studies relative to the diversity in anatomy, pathology, and acquisition conditions. While multimodal imaging provides complementary information critical for clinical interpretation, effectively integrating these signals remains difficult. We propose Multimodal Intra- and Cross-Context Vision Transformer (MICViT), a 3D vision transformer that explicitly models both modality-specific representations and cross-modal interactions across local and global contexts. Concretely, MICViT combines four attention mechanisms: modality-specific local and global attention for intra-modal feature learning, and cross-modal local and global attention to capture interactions between modalities. We evaluate MICViT on brain age prediction across three heterogeneous datasets (UK Biobank, n=41,404; SOOP, n=1,062; Cam-CAN, n=613) using multiple MRI modalities (e.g. T1, FLAIR, DWI, SWI). MICViT consistently outperforms state-of-the-art CNN and transformer baselines in 3D settings. Notably, it benefits more strongly from multimodal inputs, yielding larger performance gains as additional modalities are incorporated. These results demonstrate that explicitly modeling intra- and cross-modal interactions is key to unlocking the full potential of multimodal brain MRI, highlighting a promising direction for representation learning in neuroimaging.

Problem

• multimodal MRI
• 3D brain imaging
• cross-modal context
• limited sample size
• neuroimaging representation learning

Innovation

• multimodal learning
• 3D vision transformer
• cross-modal attention
• brain MRI
• context modeling