Towards unified brain-to-text decoding across speech production and perception

2026-03-12

AI Summary

This study addresses the limitations of existing brain-to-text decoding approaches, which are largely confined to single modalities (production or perception) and alphabetic scripts, thereby struggling with the structural complexity of Chinese characters and cross-character generalization. To overcome these challenges, this work proposes the first unified multimodal brain-decoding framework for Mandarin, explicitly driven by syllabic structure and simultaneously supporting both speech production and perception. The method integrates neural signal–guided initial/final phoneme classification, a three-stage post-training procedure leveraging a 7-billion-parameter large language model, and a two-stage inference architecture, enabling accurate sentence-level decoding of out-of-vocabulary characters. The framework surpasses commercial models with over ten billion parameters in performance and reveals key neurocognitive mechanisms: speech production engages broader cortical regions, while perception exhibits a temporal lag despite shared neural activity patterns across modalities.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Speech production and perception are the main ways humans communicate daily. Prior brain-to-text decoding studies have largely focused on a single modality and alphabetic languages. Here, we present a unified brain-to-sentence decoding framework for both speech production and perception in Mandarin Chinese. The framework exhibits strong generalization ability, enabling sentence-level decoding when trained only on single-character data and supporting characters and syllables unseen during training. In addition, it allows direct and controlled comparison of neural dynamics across modalities. Mandarin speech is decoded by first classifying syllable components in Hanyu Pinyin, namely initials and finals, from neural signals, followed by a post-trained large language model (LLM) that maps sequences of toneless Pinyin syllables to Chinese sentences. To enhance LLM decoding, we designed a three-stage post-training and two-stage inference framework based on a 7-billion-parameter LLM, achieving overall performance that exceeds larger commercial LLMs with hundreds of billions of parameters or more. In addition, several characteristics were observed in Mandarin speech production and perception: speech production involved neural responses across broader cortical regions than auditory perception; channels responsive to both modalities exhibited similar activity patterns, with speech perception showing a temporal delay relative to production; and decoding performance was broadly comparable across hemispheres. Our work not only establishes the feasibility of a unified decoding framework but also provides insights into the neural characteristics of Mandarin speech production and perception. These advances contribute to brain-to-text decoding in logosyllabic languages and pave the way toward neural language decoding systems supporting multiple modalities.

Problem

• brain-to-text decoding
• speech production
• speech perception
• Mandarin Chinese
• unified framework

Innovation

• brain-to-text decoding
• unified framework
• Mandarin Chinese
• large language model (LLM)
• speech production and perception