What Does the Brain See? Multiview Neural Representations to Demystify the Brain-Visual Alignment

2026-06-24

AI Summary

This work addresses the challenge of zero-shot visual semantic decoding from electroencephalography (EEG) signals, which are inherently low signal-to-noise ratio, non-stationary, and spatially limited. The authors propose a unified multi-view EEG representation learning framework that jointly models temporal dynamics, adaptive spectral decomposition, and structured inter-electrode interactions for the first time. By integrating a conditional state-space model, learnable wavelet transforms, an attention-based graph neural network, and contrastive learning with EEG-specific regularization, the framework aligns multi-view EEG embeddings with pretrained visual representations in a shared semantic space. The study introduces a novel cross-session EEG-to-image decoding evaluation paradigm and achieves state-of-the-art performance on the THINGS-EEG benchmark, with within-subject (Top-1/5: 54.8%/85.6%), cross-subject (15.3%/45.4%), and cross-session (40.8%/78.0%) accuracies, substantially advancing semantic alignment and generalization capabilities.

Abstract

Zero-shot visual decoding from electroencephalography (EEG) aims to infer visual semantics from non-invasive neural recordings, but remains challenging due to the low signal-to-noise ratio, non-stationarity, and limited spatial resolution of EEG. Existing EEG-vision alignment methods often rely on holistic EEG embeddings, which can obscure the complementary temporal, spectral, and spatial structure underlying visual perception. We introduce a unified multiview EEG representation learning framework for aligning brain responses with visual semantic embeddings. Our method builds an EEG encoder that jointly models three complementary views: input-conditioned state-space temporal dynamics, learnable wavelet-based spectral decomposition for sample-adaptive frequency modeling, and attention-modulated graph learning for structured electrode interactions. The resulting multiview EEG embeddings are fused and aligned with pretrained visual representations in a shared semantic space using contrastive learning with EEG-specific regularization, enabling 200-way zero-shot visual classification. Experiments on THINGS-EEG benchmark show that our method achieves state-of-the-art performance, with 54.8% Top-1 and 85.6% Top-5 accuracy in the within-subject setting and 15.3% Top-1 and 45.4% Top-5 accuracy in the cross-subject setting. We further present the first systematic cross-session EEG-image decoding evaluation, achieving 40.8% Top-1 and 78.0% Top-5 accuracy. These results suggest that explicitly modeling multiview neural structure improves both semantic alignment and generalization in EEG-based visual decoding.

Problem

• EEG
• visual decoding
• zero-shot classification
• brain-visual alignment
• neural representation

Innovation

• multiview representation learning
• EEG-visual alignment
• zero-shot visual decoding
• contrastive learning
• structured electrode interactions