Brain-DiT: A Universal Multi-state fMRI Foundation Model with Metadata-Conditioned Pretraining

2026-04-14

AI Summary

Existing fMRI foundation models struggle to learn generalizable representations due to limited coverage of brain states and mismatched pretraining objectives. This work proposes the first metadata-conditioned diffusion pretraining framework, leveraging a Diffusion Transformer to model raw fMRI signals directly. By integrating 349,898 sessions across 24 diverse datasets—including resting-state, task-based, naturalistic stimulation, clinical, and sleep conditions—the model disentangles neural dynamics from population-level variability and enables multiscale representation learning. Evaluated on seven downstream tasks, the approach significantly outperforms reconstruction- or alignment-based baselines. Notably, the learned representations reveal scale-dependent task preferences: for instance, ADNI-based classification relies more on global semantic features, whereas age and gender prediction benefit predominantly from local structural patterns.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Diffusion Models for Vision
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Current fMRI foundation models primarily rely on a limited range of brain states and mismatched pretraining tasks, restricting their ability to learn generalized representations across diverse brain states. We present \textit{Brain-DiT}, a universal multi-state fMRI foundation model pretrained on 349,898 sessions from 24 datasets spanning resting, task, naturalistic, disease, and sleep states. Unlike prior fMRI foundation models that rely on masked reconstruction in the raw-signal space or a latent space, \textit{Brain-DiT} adopts metadata-conditioned diffusion pretraining with a Diffusion Transformer (DiT), enabling the model to learn multi-scale representations that capture both fine-grained functional structure and global semantics. Across extensive evaluations and ablations on 7 downstream tasks, we find consistent evidence that diffusion-based generative pretraining is a stronger proxy than reconstruction or alignment, with metadata-conditioned pretraining further improving downstream performance by disentangling intrinsic neural dynamics from population-level variability. We also observe that downstream tasks exhibit distinct preferences for representational scale: ADNI classification benefits more from global semantic representations, whereas age/sex prediction comparatively relies more on fine-grained local structure. Code and parameters of Brain-DiT are available at \href{https://github.com/REDMAO4869/Brain-DiT}{Link}.

Problem

• fMRI foundation model
• multi-state brain representation
• generalized neural representation
• brain state diversity
• pretraining task mismatch

Innovation

• diffusion pretraining
• metadata-conditioned
• multi-state fMRI
• Diffusion Transformer
• foundation model