Improving Multimodal Brain Encoding Model with Dynamic Subject-awareness Routing

2025-10-06

AI Summary

Naturalistic fMRI encoding faces challenges including heterogeneous multimodal inputs, static fusion mechanisms, and substantial inter-subject variability. To address these, we propose AFIRE-MIND: AFIRE standardizes multimodal alignment and fusion to unify heterogeneous feature representations; MIND, a subject-prior-guided decoder, incorporates dynamic gating and Top-K sparse expert routing for content-aware, subject-adaptive mixture modeling. Our framework decouples fusion and decoding design, enabling end-to-end training and plug-and-play extensibility. Evaluated across multiple multimodal backbones and subject cohorts, AFIRE-MIND significantly outperforms strong baselines in whole-brain response prediction accuracy and cross-subject generalization. Furthermore, expert activation patterns exhibit interpretability and strong correlation with stimulus semantic categories.

Technology Category

• Machine Learning: Multimodal Learning
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Naturalistic fMRI encoding must handle multimodal inputs, shifting fusion styles, and pronounced inter-subject variability. We introduce AFIRE (Agnostic Framework for Multimodal fMRI Response Encoding), an agnostic interface that standardizes time-aligned post-fusion tokens from varied encoders, and MIND, a plug-and-play Mixture-of-Experts decoder with a subject-aware dynamic gating. Trained end-to-end for whole-brain prediction, AFIRE decouples the decoder from upstream fusion, while MIND combines token-dependent Top-K sparse routing with a subject prior to personalize expert usage without sacrificing generality. Experiments across multiple multimodal backbones and subjects show consistent improvements over strong baselines, enhanced cross-subject generalization, and interpretable expert patterns that correlate with content type. The framework offers a simple attachment point for new encoders and datasets, enabling robust, plug-and-improve performance for naturalistic neuroimaging studies.

Problem

• Handling multimodal inputs and shifting fusion styles
• Addressing pronounced inter-subject variability in fMRI
• Improving cross-subject generalization with personalized expert usage

Innovation

• Standardizes time-aligned tokens from varied encoders
• Uses subject-aware dynamic gating for expert routing
• Combines Top-K sparse routing with subject prior