NeuroBOLT: Resting-state EEG-to-fMRI Synthesis with Multi-dimensional Feature Mapping

2024-10-07

Neural Information Processing Systems

AI Summary

This study addresses the high cost and immobility of fMRI, as well as the limitations of existing EEG-to-fMRI mapping approaches—namely, their restriction to a few cortical regions and single brain states. We propose the first end-to-end EEG-to-fMRI synthesis framework capable of whole-brain (including cortex, higher-order cognitive areas, and deep subcortical nuclei) and cross-condition (e.g., resting-state generalization) signal reconstruction. Methodologically, we introduce the Neuro-to-BOLD Transformer, which integrates multi-scale spatio-temporal-spectral feature encoding, spectrogram-guided spatial attention, joint temporal convolutional and graph neural network modeling, and a novel cross-modal alignment loss. Evaluated on multiple public datasets, our framework achieves state-of-the-art performance, enabling, for the first time, high-fidelity whole-brain resting-state BOLD signal reconstruction. It significantly outperforms prior methods in key regions—including the default mode network and thalamus—overcoming both EEG’s spatial ambiguity and the challenges of accurate BOLD response modeling.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Multimodal Learning
• Humans and AI: Brain-Sensing and Analysis

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Functional magnetic resonance imaging (fMRI) is an indispensable tool in modern neuroscience, providing a non-invasive window into whole-brain dynamics at millimeter-scale spatial resolution. However, fMRI is constrained by issues such as high operation costs and immobility. With the rapid advancements in cross-modality synthesis and brain decoding, the use of deep neural networks has emerged as a promising solution for inferring whole-brain, high-resolution fMRI features directly from electroencephalography (EEG), a more widely accessible and portable neuroimaging modality. Nonetheless, the complex projection from neural activity to fMRI hemodynamic responses and the spatial ambiguity of EEG pose substantial challenges both in modeling and interpretability. Relatively few studies to date have developed approaches for EEG-fMRI translation, and although they have made significant strides, the inference of fMRI signals in a given study has been limited to a small set of brain areas and to a single condition (i.e., either resting-state or a specific task). The capability to predict fMRI signals in other brain areas, as well as to generalize across conditions, remain critical gaps in the field. To tackle these challenges, we introduce a novel and generalizable framework: NeuroBOLT, i.e., Neuro-to-BOLD Transformer, which leverages multi-dimensional representation learning from temporal, spatial, and spectral domains to translate raw EEG data to the corresponding fMRI activity signals across the brain. Our experiments demonstrate that NeuroBOLT effectively reconstructs unseen resting-state fMRI signals from primary sensory, high-level cognitive areas, and deep subcortical brain regions, achieving state-of-the-art accuracy with the potential to generalize across varying conditions and sites, which significantly advances the integration of these two modalities.

Problem

• Inferring whole-brain fMRI from EEG data
• Overcoming spatial ambiguity in EEG signals
• Generalizing fMRI prediction across brain conditions

Innovation

• Leverages multi-dimensional temporal spatial spectral representation learning
• Translates raw EEG data to fMRI signals across entire brain
• Generalizes across varying conditions and sites with state-of-art accuracy