Can LLMs Understand Unvoiced Speech? Exploring EMG-to-Text Conversion with LLMs

2025-05-30

AI Summary

Training EMG-to-text models for aphasic individuals is hindered by the absence of paired speech data. Method: We propose the first LLM-adaptation framework for silent electromyography (sEMG)—i.e., sEMG recorded without audible vocalization—featuring a lightweight EMG adapter that directly maps surface EMG features into the embedding space of a large language model (LLM), eliminating reliance on spoken audio or paired data. Our end-to-end adaptation integrates sEMG signal processing, feature-space alignment, and embedding-space transfer learning. Results: On closed-vocabulary silent sEMG-to-text decoding, our method achieves a mean word error rate of 0.49%; it surpasses existing specialized models by ~20% using only six minutes of sEMG data. This work provides the first empirical validation that LLMs can directly interpret biological speech signals, establishing a novel paradigm for silent speech decoding.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Speech
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Unvoiced electromyography (EMG) is an effective communication tool for individuals unable to produce vocal speech. However, most prior methods rely on paired voiced and unvoiced EMG signals, along with speech data, for EMG-to-text conversion, which is not practical for such individuals. Given the rise of large language models (LLMs) in speech recognition, we explore their potential to understand unvoiced speech. To this end, we address the challenge of learning from unvoiced EMG alone and propose a novel EMG adaptor module that maps EMG features into an LLM's input space, achieving an average word error rate (WER) of 0.49 on a closed-vocabulary unvoiced EMG-to-text task. Even with a conservative data availability of just six minutes, our approach improves performance over specialized models by nearly 20%. While LLMs have been shown to be extendable to new language modalities -- such as audio -- understanding articulatory biosignals like unvoiced EMG remains more challenging. This work takes a crucial first step toward enabling LLMs to comprehend unvoiced speech using surface EMG.

Problem

• Exploring EMG-to-text conversion using LLMs for unvoiced speech
• Addressing challenge of learning from unvoiced EMG signals alone
• Proposing EMG adaptor module to map EMG features for LLMs

Innovation

• EMG adaptor module maps EMG to LLM input
• Achieves 0.49 WER on EMG-to-text task
• Improves performance with minimal training data