An Empirical Analysis of Discrete Unit Representations in Speech Language Modeling Pre-training

2025-09-03

AI Summary

This study addresses the optimization of discrete speech unit representations in Speech Language Models (SLMs) to enhance speech modeling during continual pretraining. We propose a systematic framework encompassing speech encoder design, controllable clustering granularity, phoneme-aligned discretization, and domain-matched data selection. We first uncover a strong correlation between model scale and optimal clustering granularity, and demonstrate that discrete vocabularies explicitly encode both linguistic and paralinguistic structures. Experiments show that principled discretization yields substantial downstream improvements—averaging +2.1% relative WER reduction—and that distributional alignment between clustering data and target domains enhances model robustness and generalization. Our work provides a reproducible methodology and theoretical insights for efficient speech representation learning in SLMs.

Technology Category

• Natural Language Processing: Speech
• Search and Optimization: Mixed Discrete/Continuous Search
• Machine Learning: Representation Learning

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

This paper investigates discrete unit representations in Speech Language Models (SLMs), focusing on optimizing speech modeling during continual pre-training. In this paper, we systematically examine how model architecture, data representation, and training robustness influence the pre-training stage in which we adapt existing pre-trained language models to the speech modality. Our experiments highlight the role of speech encoders and clustering granularity across different model scales, showing how optimal discretization strategies vary with model capacity. By examining cluster distribution and phonemic alignments, we investigate the effective use of discrete vocabulary, uncovering both linguistic and paralinguistic patterns. Additionally, we explore the impact of clustering data selection on model robustness, highlighting the importance of domain matching between discretization training and target applications.

Problem

• Optimizing discrete unit representations for speech language model pre-training
• Examining how model architecture and data representation affect speech adaptation
• Investigating optimal discretization strategies across different model capacities

Innovation

• Discrete unit representations optimize speech modeling
• Examine model architecture, data representation, training robustness
• Speech encoders and clustering granularity across scales