Estimating Brain Activity with High Spatial and Temporal Resolution using a Naturalistic MEG-fMRI Encoding Model

2025-10-10

AI Summary

Non-invasive neuroimaging faces a fundamental spatiotemporal resolution trade-off: MEG offers millisecond temporal resolution but coarse spatial localization, whereas fMRI provides millimeter-scale spatial precision at the cost of slow temporal sampling; existing single-trial source imaging methods fail to achieve concurrent millisecond and millimeter accuracy. We propose a Transformer-based multimodal encoding model that, for the first time, jointly models cross-subject MEG and fMRI data under natural speech stimulation, with latent representations directly corresponding to high spatiotemporal-resolution cortical source activity. This unified framework transcends conventional source localization and modality fusion paradigms. In simulation experiments, it significantly outperforms the minimum-norm estimate and unimodal baselines. Critically, the estimated source signals exhibit strong cross-subject generalizability and—on independent ECoG data—even surpass dedicated ECoG models, establishing a novel tool for fine-grained mechanistic investigation of natural cognition.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Current non-invasive neuroimaging techniques trade off between spatial resolution and temporal resolution. While magnetoencephalography (MEG) can capture rapid neural dynamics and functional magnetic resonance imaging (fMRI) can spatially localize brain activity, a unified picture that preserves both high resolutions remains an unsolved challenge with existing source localization or MEG-fMRI fusion methods, especially for single-trial naturalistic data. We collected whole-head MEG when subjects listened passively to more than seven hours of narrative stories, using the same stimuli in an open fMRI dataset (LeBel et al., 2023). We developed a transformer-based encoding model that combines the MEG and fMRI from these two naturalistic speech comprehension experiments to estimate latent cortical source responses with high spatiotemporal resolution. Our model is trained to predict MEG and fMRI from multiple subjects simultaneously, with a latent layer that represents our estimates of reconstructed cortical sources. Our model predicts MEG better than the common standard of single-modality encoding models, and it also yields source estimates with higher spatial and temporal fidelity than classic minimum-norm solutions in simulation experiments. We validated the estimated latent sources by showing its strong generalizability across unseen subjects and modalities. Estimated activity in our source space predict electrocorticography (ECoG) better than an ECoG-trained encoding model in an entirely new dataset. By integrating the power of large naturalistic experiments, MEG, fMRI, and encoding models, we propose a practical route towards millisecond-and-millimeter brain mapping.

Problem

• Resolving trade-off between spatial and temporal resolution in neuroimaging
• Integrating MEG and fMRI data for unified brain activity mapping
• Estimating cortical source responses from naturalistic speech comprehension experiments

Innovation

• Transformer model integrates MEG and fMRI data
• Latent layer reconstructs high-resolution cortical sources
• Validated with cross-subject and cross-modality generalization