Learning Sparse Latent Predictive Foundation Model for Multimodal Neuroimaging

2026-06-12

AI Summary

Existing approaches struggle to uniformly model multi-contrast brain MRI (e.g., T1w, T2w, FLAIR) at the scale of clinical healthcare systems. This work proposes Neuro-JEPA, the first framework to introduce the sparse latent predictive modeling paradigm to multimodal neuroimaging. By integrating a Mixture-of-Experts architecture, a multimodal masking strategy, and a large-scale preprocessing pipeline, the study systematically investigates the impact of architectural design, masking schemes, objective functions, and sparsity on representation learning. Evaluated across 25 tasks from three medical institutions and 22 tasks on 12 public datasets, Neuro-JEPA substantially outperforms current neuroimaging foundation models and CNN baselines, demonstrating superior performance and robustness.

Abstract

Brain MRIs are routinely acquired as multiple complementary sequences with unique contrast weighting, including T1-weighed imaging (T1w) anatomic and fluid-sensitive T2-weighted (T2w) contrasts. However, methods for learning unified representations across the multitude of MRI contrast mechanisms at health-system scale are lacking. In this study, we introduce Neuro-JEPA, a sparse multimodal neuroimaging foundation model that combines a latent predictive objective with a Mixture-of-Experts architecture to encode brain MRI across core T1w, T2w, and fluid-suppressed FLAIR imaging (FLAIR). We further provide a systematic methodological study of architectural, masking, objective, and sparsity design choices beneficial for robust neuroimaging multimodal representation learning. Neuro-JEPA was pretrained on 1,551,862 scans from 428,647 studies after modality-specific preprocessing with data curation across three core structural brain MRI sequences. We evaluated the learned representations across clinical and research settings, including 25 tasks from three health systems: NYU Langone, NYU Long Island, and Massachusetts General Hospital, and 22 tasks from 12 public datasets, covering unimodal, multimodal and cross-domain evaluation configurations. Across these benchmarks, existing neuroimaging foundation models showed inconsistent gains over a simple convolutional neural network (CNN) baseline, whereas Neuro-JEPA achieved stronger and more consistent performance across all evaluated settings. These results establish a scalable methodological framework for multimodal neuroimaging representation learning and highlight the need for foundation model evaluation protocols that include simple baselines, clinically heterogeneous cohorts and controlled multimodal comparisons.

Problem

• multimodal neuroimaging
• foundation model
• MRI contrast mechanisms
• representation learning
• scalable framework

Innovation

• sparse multimodal foundation model
• latent predictive learning
• Mixture-of-Experts
• neuroimaging representation learning
• multimodal MRI