EEGPT: Unleashing the Potential of EEG Generalist Foundation Model by Autoregressive Pre-training

2024-10-14

arXiv.org

AI Summary

Current EEG research is hindered by heterogeneous data formats, outdated pretraining paradigms (e.g., masked autoencoding), and limited transferability, impeding the development of general-purpose foundation models. To address this, we propose EEGPT—the first general-purpose foundation model for EEG. Our approach introduces: (1) electrode-level sequence modeling with autoregressive pretraining, replacing conventional masked autoencoding; (2) a scalable electrode graph neural network architecture supporting arbitrary combinations of up to 138 channels, enabling seamless adaptation across multi-device, multi-subject, and multi-task scenarios; and (3) large-scale parameter scaling (up to 1.1B parameters) and unified evaluation across 12 public benchmarks and 5 downstream task categories, where EEGPT consistently outperforms task-specific models. Ablation studies validate the efficacy of each component. Code and pretrained models will be publicly released.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Electroencephalogram (EEG) signals are pivotal in providing insights into spontaneous brain activity, highlighting their significant importance in neuroscience research. However, the exploration of versatile EEG models is constrained by diverse data formats, outdated pre-training paradigms, and limited transfer learning methods, only leading to specialist models on single dataset. In this paper, we introduce EEGPT, the first generalist EEG foundation model designed to address these challenges. First, we propose an electrode-wise modeling strategy that treats each electrode as a fundamental unit, enabling the integration of diverse EEG datasets collected from up to 138 electrodes, amassing 37.5M pre-training samples. Second, we develop the first autoregressive EEG pre-trained model, moving away from traditional masked autoencoder approaches to a next signal prediction task that better captures the sequential and temporal dependencies of EEG data. We also explore scaling laws with model up to 1.1B parameters: the largest in EEG research to date. Third, we introduce a multi-task transfer learning paradigm using a learnable electrode graph network shared across tasks, which for the first time confirms multi-task compatibility and synergy. As the first generalist EEG foundation model, EEGPT shows broad compatibility with various signal acquisition devices, subjects, and tasks. It supports up to 138 electrodes and any combination thereof as input. Furthermore, we simultaneously evaluate it on 5 distinct tasks across 12 benchmarks. EEGPT consistently outperforms existing specialist models across all downstream tasks, with its effectiveness further validated through extensive ablation studies. This work sets a new direction for generalist EEG modeling, offering improved scalability, transferability, and adaptability for a wide range of EEG applications. The code and models will be released.

Problem

• Develops first generalist EEG foundation model overcoming dataset and format limitations
• Introduces autoregressive pre-training for EEG signals capturing temporal dependencies
• Enables multi-task compatibility across diverse EEG acquisition devices and subjects

Innovation

• Electrode-wise modeling strategy for diverse EEG datasets
• Autoregressive pre-training for sequential EEG dependencies
• Multi-task transfer learning with electrode graph network