SUP-MCRL: Subject-aware Unified Pseudo-feature Coded Multimodal Contrastive Representation Learning for EEG Visual Decoding

2026-06-15

AI Summary

This study addresses the significant drop in neural decoding fidelity of non-invasive brain–computer interfaces under naturalistic visual conditions, primarily caused by existing methods neglecting semantic consistency and subject-specific characteristics, which leads to failure in zero-shot alignment. To overcome this, the authors propose a unified framework, SUP-MCRL, which integrates a Semantic Entity-aware Visual Encoder (SAVE), a Unified EEG Enhancer (UEE) driven by multi-scale dilated convolutions and inter-band attention, and a Prototype Progressive Augmentation (PPA)-based contrastive learning mechanism. This approach jointly optimizes geometric alignment and semantic consistency of cross-modal representations. Evaluated on the THINGS-EEG dataset, the method achieves within-subject Top-1/Top-5 accuracies of 66.0%/91.9% and leave-one-subject-out zero-shot accuracies of 24.0%/52.9%, substantially outperforming current state-of-the-art approaches.

Abstract

Non-invasive brain-computer interfaces suffer severe fidelity degradation in neural visual decoding when generalizing to natural visual experiences. Conventional multimodal contrastive representation learning solely optimizes geometric distance alignment, neglecting semantic consistency and subject selectivity, causing spurious zero-shot alignment. We propose SUP-MCRL, a unified framework integrating three collaborative mechanisms: (1) Semantic-entity Aware Visual Encoder (SAVE), learning spatial attention to extract semantic content without pre-trained saliency models; (2 Unified EEG Enhancer (UEE), employing multi-scale atrous convolutions and inter-band attention for adaptive cross-subject robustness; and (3) Prototype-based Progressive Augmenter (PPA), maintaining an EMA-updated pseudo-feature pool to prevent representation collapse. Zero-shot experiments on THINGS-EEG achieve 66.0%/91.9% (Top-1/Top-5) intra-subject and 24.0%/52.9% LOSO accuracy, surpassing state-of-the-art methods. Code is available at https://github.com/NZWANG/SUP-MCRL.

Problem

• EEG visual decoding
• multimodal contrastive learning
• subject selectivity
• semantic consistency
• zero-shot alignment

Innovation

• multimodal contrastive learning
• EEG visual decoding
• subject-aware representation
• semantic alignment
• zero-shot generalization