Network-Specific Models for Multimodal Brain Response Prediction

2025-07-25

arXiv.org

AI Summary

This study addresses whole-brain response prediction under complex multimodal movie stimuli. We propose a functional-network-clustering–based modeling paradigm: the brain is parcellated into four functionally coherent clusters using the Yeo 7-network atlas; for each cluster, a dedicated multi-subject MLP is trained with adaptive temporal dynamics and modality-weighting mechanisms to capture network-specific spatiotemporal properties and differential modality contributions. Fine-grained cortical parcellation is achieved via the Schaefer atlas, enabling intra-cluster parameter optimization and adaptive cross-temporal memory updating. Evaluated in the Algonauts Project 2025 challenge, our method ranks 8th overall and achieves out-of-distribution (OOD) generalization correlation 1.9× higher than the baseline. It significantly improves prediction accuracy across >1,000 brain regions, demonstrating the efficacy of functional-prioritized modeling for multimodal neural decoding.

Technology Category

• Machine Learning: Multimodal Learning
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

In this work, we present a network-specific approach for predicting brain responses to complex multimodal movies, leveraging the Yeo 7-network parcellation of the Schaefer atlas. Rather than treating the brain as a homogeneous system, we grouped the seven functional networks into four clusters and trained separate multi-subject, multi-layer perceptron (MLP) models for each. This architecture supports cluster-specific optimization and adaptive memory modeling, allowing each model to adjust temporal dynamics and modality weighting based on the functional role of its target network. Our results demonstrate that this clustered strategy significantly enhances prediction accuracy across the 1,000 cortical regions of the Schaefer atlas. The final model achieved an eighth-place ranking in the Algonauts Project 2025 Challenge, with out-of-distribution (OOD) correlation scores nearly double those of the baseline model used in the selection phase. Code is available at https://github.com/Corsi01/algo2025.

Problem

• Predict brain responses to multimodal movies using deep learning
• Cluster functional networks for specialized prediction models
• Enhance accuracy across cortical regions with adaptive modeling

Innovation

• Grouped brain networks into four functional clusters
• Trained separate MLP models for each cluster
• Used adaptive memory modeling for temporal dynamics