Decoding Covert Speech from EEG Using a Functional Areas Spatio-Temporal Transformer

2025-04-02

AI Summary

This study addresses the challenge of EEG-based decoding of covert speech—hindered by poorly understood neuroarticulatory mapping mechanisms and low EEG signal-to-noise ratio. To this end, we introduce the first large-scale, synchronized EEG dataset comprising both covert and overt speech from 57 participants, covering multi-word utterances. We propose the Functional-Area Spatio-Temporal Transformer (FAST), the first model to integrate brain functional parcellation priors for guided time-frequency feature extraction, channel-adaptive weighting, and spatio-temporal Transformer encoding—enabling high-accuracy, character-level sequential decoding. Experiments reveal functionally specific activation patterns in frontal and temporal regions during covert speech, with interpretable visualizations offering novel neuroscientific evidence. FAST achieves state-of-the-art performance while maintaining model interpretability. The code is publicly released, advancing both brain–computer interface development and neural language decoding research.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Natural Language Processing: Speech
• Machine Learning: Mixture of Experts (MoE)

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Security and Privacy: Large-scale security measurements

Abstract

Covert speech involves imagining speaking without audible sound or any movements. Decoding covert speech from electroencephalogram (EEG) is challenging due to a limited understanding of neural pronunciation mapping and the low signal-to-noise ratio of the signal. In this study, we developed a large-scale multi-utterance speech EEG dataset from 57 right-handed native English-speaking subjects, each performing covert and overt speech tasks by repeating the same word in five utterances within a ten-second duration. Given the spatio-temporal nature of the neural activation process during speech pronunciation, we developed a Functional Areas Spatio-temporal Transformer (FAST), an effective framework for converting EEG signals into tokens and utilizing transformer architecture for sequence encoding. Our results reveal distinct and interpretable speech neural features by the visualization of FAST-generated activation maps across frontal and temporal brain regions with each word being covertly spoken, providing new insights into the discriminative features of the neural representation of covert speech. This is the first report of such a study, which provides interpretable evidence for speech decoding from EEG. The code for this work has been made public at https://github.com/Jiang-Muyun/FAST

Problem

• Decoding covert speech from EEG signals
• Overcoming low signal-to-noise ratio in EEG
• Mapping neural activation during speech pronunciation

Innovation

• Functional Areas Spatio-temporal Transformer (FAST) framework
• EEG signals converted into transformer-compatible tokens
• Visualization of brain activation maps for interpretability