Dynamic Vision from EEG Brain Recordings: How much does EEG know?

2025-05-27

AI Summary

EEG signals are inherently non-stationary, exhibit low signal-to-noise ratios, and suffer from severe scarcity of dynamic visual stimulation data—making video reconstruction substantially more challenging than static image reconstruction. To address this, we propose the first end-to-end EEG-to-video generation framework: (1) a triplet contrastive learning module to extract robust, discriminative neural features; (2) an enhanced StyleGAN-ADA architecture incorporating temporal conditioning mechanisms to explicitly model video dynamics; and (3) integrated cross-modal alignment and source localization analysis to uncover functional specificity—revealing that occipital and parietal regions differentially encode motion direction and temporal rhythm. Evaluated on our newly constructed EEG-Video dataset, our method achieves frame-level perceptually identifiable video reconstruction. This work provides the first quantitative evidence that EEG encodes rich dynamic visual representations, establishing a novel paradigm and benchmark for brain–computer vision interfaces and neural decoding.

Technology Category

• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Multimodal Learning

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Reconstructing and understanding dynamic visual information (video) from brain EEG recordings is challenging due to the non-stationary nature of EEG signals, their low signal-to-noise ratio (SNR), and the limited availability of EEG-Video stimulus datasets. Most recent studies have focused on reconstructing static images from EEG recordings. In this work, we propose a framework to reconstruct dynamic visual stimuli from EEG data and conduct an in-depth study of the information encoded in EEG signals. Our approach first trains a feature extraction network using a triplet-based contrastive learning strategy within an EEG-video generation framework. The extracted EEG features are then used for video synthesis with a modified StyleGAN-ADA, which incorporates temporal information as conditioning. Additionally, we analyze how different brain regions contribute to processing dynamic visual stimuli. Through several empirical studies, we evaluate the effectiveness of our framework and investigate how much dynamic visual information can be inferred from EEG signals. The inferences we derive through our extensive studies would be of immense value to future research on extracting visual dynamics from EEG.

Problem

• Reconstructing dynamic videos from noisy EEG signals
• Analyzing EEG-encoded visual information across brain regions
• Developing EEG-to-video synthesis using contrastive learning

Innovation

• Triplet-based contrastive learning for EEG feature extraction
• Modified StyleGAN-ADA for EEG-to-video synthesis
• Analysis of brain region contributions to dynamic vision