StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs

2025-09-26

AI Summary

Current semantic speech tokenizers exhibit high sensitivity to meaningless acoustic perturbations, yielding unstable outputs even under high signal-to-noise ratios—thereby significantly increasing the learning burden on downstream SpeechLLMs. To address this, we propose a consensus-driven multi-branch tokenization architecture that employs parallel encoding paths and a bit-level bitwise voting mechanism to mitigate the fragility of single-path quantization and alleviate sparse training signal issues. Integrated with targeted optimization strategies, our model achieves stable and robust semantic tokenization across diverse noise conditions. Experiments demonstrate that the proposed method substantially reduces the Unit Edit Distance (UED) to the state-of-the-art level. Furthermore, it markedly enhances the noise robustness and generalization capability of SpeechLLMs on downstream tasks—including ASR and speech understanding—without architectural modifications to the LMs themselves.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Prevalent semantic speech tokenizers, designed to capture linguistic content, are surprisingly fragile. We find they are not robust to meaning-irrelevant acoustic perturbations; even at high Signal-to-Noise Ratios (SNRs) where speech is perfectly intelligible, their output token sequences can change drastically, increasing the learning burden for downstream LLMs. This instability stems from two flaws: a brittle single-path quantization architecture and a distant training signal indifferent to intermediate token stability. To address this, we introduce StableToken, a tokenizer that achieves stability through a consensus-driven mechanism. Its multi-branch architecture processes audio in parallel, and these representations are merged via a powerful bit-wise voting mechanism to form a single, stable token sequence. StableToken sets a new state-of-the-art in token stability, drastically reducing Unit Edit Distance (UED) under diverse noise conditions. This foundational stability translates directly to downstream benefits, significantly improving the robustness of SpeechLLMs on a variety of tasks.

Problem

• Addresses fragility of semantic speech tokenizers to acoustic noise
• Improves token stability under diverse noise conditions
• Enhances robustness of SpeechLLMs for various tasks

Innovation

• Multi-branch architecture processes audio in parallel
• Bit-wise voting merges representations for stable tokens
• Consensus-driven mechanism enhances noise robustness