Brain-to-Text Decoding: A Non-invasive Approach via Typing

2025-02-18

AI Summary

This study addresses the need for non-invasive neural decoding in individuals with aphasia or motor impairments. We propose a novel sentence-level brain signal decoding paradigm—“covert rehearsal–tapping”—and introduce Brain2Qwerty, an end-to-end deep learning architecture that jointly processes MEG and EEG signals to directly predict character sequences. Our contributions are threefold: (1) we establish the first non-invasive sentence decoding paradigm anchored on typing behavior; (2) we demonstrate that decoding performance is synergistically driven by both motor execution and high-level language cognition; and (3) we infer underlying cognitive mechanisms via systematic error pattern analysis. Evaluation on healthy participants shows a mean character error rate (CER) of 32% for MEG (best individual: 19%), substantially outperforming EEG (67% CER) and significantly narrowing the performance gap with invasive approaches.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Humans and AI: Brain-Sensing and Analysis
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Security and Privacy: Large-scale security measurements

Abstract

Modern neuroprostheses can now restore communication in patients who have lost the ability to speak or move. However, these invasive devices entail risks inherent to neurosurgery. Here, we introduce a non-invasive method to decode the production of sentences from brain activity and demonstrate its efficacy in a cohort of 35 healthy volunteers. For this, we present Brain2Qwerty, a new deep learning architecture trained to decode sentences from either electro- (EEG) or magneto-encephalography (MEG), while participants typed briefly memorized sentences on a QWERTY keyboard. With MEG, Brain2Qwerty reaches, on average, a character-error-rate (CER) of 32% and substantially outperforms EEG (CER: 67%). For the best participants, the model achieves a CER of 19%, and can perfectly decode a variety of sentences outside of the training set. While error analyses suggest that decoding depends on motor processes, the analysis of typographical errors suggests that it also involves higher-level cognitive factors. Overall, these results narrow the gap between invasive and non-invasive methods and thus open the path for developing safe brain-computer interfaces for non-communicating patients.

Problem

• Non-invasive brain-to-text decoding
• Deep learning for sentence production
• Safe brain-computer interfaces development

Innovation

• Non-invasive brain-to-text decoding
• Deep learning architecture Brain2Qwerty
• Utilizes EEG and MEG data