Recent Advances in Discrete Speech Tokens: A Review

2025-02-10

AI Summary

This work addresses core challenges in adopting discrete speech tokens as fundamental acoustic representations for large language models (LLMs). It systematically surveys two dominant paradigms—acoustic tokens and semantic tokens—and establishes the first unified taxonomy. Leveraging representative models including VQ-VAE, HuBERT, WavLM, and SpeechTokenizer, the study conducts cross-paradigm comparisons grounded in information-theoretic analysis, comprehensive multi-task evaluation, and ablation studies. Results reveal that semantic tokens facilitate more effective joint modeling with LLMs, whereas acoustic tokens achieve superior waveform fidelity; yet both exhibit fundamental limitations in robustness, generalization, and text–speech alignment. To bridge this gap, the paper proposes a novel hybrid tokenization framework that jointly optimizes information capacity and controllability. This approach provides both theoretical foundations and practical guidelines for unified speech–language modeling.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

The rapid advancement of speech generation technologies in the era of large language models (LLMs) has established discrete speech tokens as a foundational paradigm for speech representation. These tokens, characterized by their discrete, compact, and concise nature, are not only advantageous for efficient transmission and storage, but also inherently compatible with the language modeling framework, enabling seamless integration of speech into text-dominated LLM architectures. Current research categorizes discrete speech tokens into two principal classes: acoustic tokens and semantic tokens, each of which has evolved into a rich research domain characterized by unique design philosophies and methodological approaches. This survey systematically synthesizes the existing taxonomy and recent innovations in discrete speech tokenization, conducts a critical examination of the strengths and limitations of each paradigm, and presents systematic experimental comparisons across token types. Furthermore, we identify persistent challenges in the field and propose potential research directions, aiming to offer actionable insights to inspire future advancements in the development and application of discrete speech tokens.

Problem

• Review advances in discrete speech tokens
• Analyze strengths and limitations of token types
• Propose future research directions for speech tokens

Innovation

• Discrete speech tokens integration
• Acoustic and semantic tokens classification
• Systematic experimental token comparisons