A foundation model with multi-variate parallel attention to generate neuronal activity

2025-06-25

AI Summary

Modeling multivariate time series—particularly intracranial electroencephalography (iEEG)—is challenging due to high inter-subject heterogeneity in channel configurations. Method: This paper introduces Multivariate Parallel Attention (MVPA), a novel attention mechanism that decouples content, temporal, and spatial attention, enabling adaptive modeling of variable-length, variable-topology channel inputs. Built upon the Transformer architecture, MVPFormer is the first generative foundation model specifically designed for neurophysiological signals, trained via large-scale self-supervised learning on iEEG datasets including SWEC, with publicly released weights. Contribution/Results: MVPFormer achieves expert-level performance in seizure detection, substantially outperforming existing Transformer-based baselines; it also sets new state-of-the-art results across diverse downstream tasks—including time-series forecasting and classification—establishing itself as the first iEEG foundation model that simultaneously demonstrates clinical validity, open accessibility, and architectural generalizability.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Learning from multi-variate time-series with heterogeneous channel configurations remains a fundamental challenge for deep neural networks (DNNs), particularly in clinical domains such as intracranial electroencephalography (iEEG), where channel setups vary widely across subjects. In this work, we introduce multi-variate parallel attention (MVPA), a novel self-attention mechanism that disentangles content, temporal, and spatial attention, enabling flexible, generalizable, and efficient modeling of time-series data with varying channel counts and configurations. We use MVPA to build MVPFormer, a generative foundation model for human electrophysiology, trained to predict the evolution of iEEG signals across diverse subjects. To support this and future effort by the community, we release the SWEC iEEG dataset, the largest publicly available iEEG dataset to date, comprising nearly 10,000 hours of recordings from heterogeneous clinical sources. MVPFormer leverages MVPA to achieve strong generalization across subjects, demonstrating expert-level performance in seizure detection and outperforming state-of-the-art Transformer baselines on our SWEC, the MAYO, and the FNUSA dataset. We further validate MVPA on standard time-series forecasting and classification tasks, where it matches or exceeds existing attention-based models. Together, our contributions establish MVPA as a general-purpose attention mechanism for heterogeneous time-series and MVPFormer as the first open-source, open-weights, and open-data iEEG foundation model with state-of-the-art clinical performance. The code is available at https://github.com/IBM/multi-variate-parallel-transformer. The SWEC iEEG dataset is available at https://mb-neuro.medical-blocks.ch/public_access/databases/ieeg/swec_ieeg.

Problem

• Handling multi-variate time-series with varying channel configurations
• Modeling iEEG signals across diverse subjects effectively
• Achieving expert-level seizure detection with generalizable attention

Innovation

• Multi-variate parallel attention mechanism
• MVPFormer for iEEG signal prediction
• Largest public SWEC iEEG dataset