BrainECHO: Semantic Brain Signal Decoding through Vector-Quantized Spectrogram Reconstruction for Whisper-Enhanced Text Generation

2024-10-19

arXiv.org

AI Summary

To address three key bottlenecks in EEG/MEG-to-text decoding—strong teacher-forcing dependency, poor cross-subject generalization, and misalignment between neural and linguistic representations—this paper proposes a decoupled three-stage framework: (i) self-supervised brain signal encoding via Mel-spectrogram discretization and VQ-VAE; (ii) unsupervised alignment of frozen latent brain features with language model representations; and (iii) teacher-forcing-free autoregressive decoding constrained by Whisper. This work is the first to jointly leverage vector-quantized spectrogram reconstruction and speech pretraining for neural decoding. Evaluated on multi-source EEG/MEG datasets, it achieves BLEU scores of 74%–89%, with a +3.65 BLEU-4 improvement over prior methods. The approach significantly enhances robustness to neural noise and cross-subject generalization, supporting sentence-level, conversational-level, and subject-agnostic decoding.

Technology Category

• Natural Language Processing: Speech
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Current EEG/MEG-to-text decoding systems suffer from three key limitations: (1) reliance on teacher-forcing methods, which compromises robustness during inference, (2) sensitivity to session-specific noise, hindering generalization across subjects, and (3) misalignment between brain signals and linguistic representations due to pre-trained language model over-dominance. To overcome these challenges, we propose BrainECHO (Brain signal decoding via vEctor-quantized speCtrogram reconstruction for WHisper-enhanced text generatiOn), a multi-stage framework that employs decoupled representation learning to achieve state-of-the-art performance on both EEG and MEG datasets. Specifically, BrainECHO consists of three stages: (1) Discrete autoencoding, which transforms continuous Mel spectrograms into a finite set of high-quality discrete representations for subsequent stages. (2) Frozen alignment, where brain signal embeddings are mapped to corresponding Mel spectrogram embeddings in a frozen latent space, effectively filtering session-specific noise through vector-quantized reconstruction, yielding a 3.65% improvement in BLEU-4 score. (3) Constrained decoding fine-tuning, which leverages the pre-trained Whisper model for audio-to-text translation, balancing signal adaptation with knowledge preservation, and achieving 74%-89% decoding BLEU scores without excessive reliance on teacher forcing. BrainECHO demonstrates robustness across sentence, session, and subject-independent conditions, passing Gaussian noise tests and showcasing its potential for enhancing language-based brain-computer interfaces.

Problem

• Overcoming teacher-forcing reliance in EEG/MEG-to-text decoding
• Reducing session-specific noise sensitivity in brain signal decoding
• Aligning brain signals with linguistic representations effectively

Innovation

• Discrete autoencoding transforms spectrograms into representations
• Frozen alignment filters noise via vector-quantized reconstruction
• Constrained decoding balances Whisper model adaptation