On the Effect of Segmentation Width and Cluster Size on Speech Resynthesis and Continuation in Generative Spoken Language Models

2026-06-22

AI Summary

This study investigates whether the performance of generative spoken language models (GSLMs) in speech synthesis and continuation is inherently constrained at low bitrates, challenging the conventional assumption that high bitrates are essential. The authors construct multi-bitrate discrete speech representations by applying fixed-width segmentation combined with K-means clustering of varying codebook sizes, and train GSLMs on these representations for evaluation. Experimental results demonstrate that even at bitrates substantially lower than baseline levels, the models consistently generate intelligible, clear, and natural-sounding speech, with continuation quality remaining stable across multiple objective metrics. The work further reveals a limited correlation between current automatic evaluation metrics and human subjective judgments, thereby affirming the feasibility and potential of low-bitrate GSLMs for practical applications.

Abstract

Generative Spoken Language Modeling (GSLM) enables text-free speech modeling by training language models (LMs) using discrete speech representations instead of textual transcription. In this paper, we investigate the performance of GSLM on speech synthesis and continuation using discrete speech representations with varying bitrates. We segment speech representations with fixed widths and train K-means models in multiple cluster sizes, resulting in various bitrate settings. We demonstrate that intelligible and natural speech can be synthesized at lower bitrate settings than the baseline. Furthermore, speech continuation quality remains stable at lower bitrates across multiple metrics, suggesting that the conventional GSLM setting may be redundant for effective speech generation. Although LLM-based metrics show higher correlation with human subjective score than conventional metrics, it remains low, highlighting the need for more stable automatic evaluation methods.

Problem

• Generative Spoken Language Modeling
• speech resynthesis
• speech continuation
• bitrate
• evaluation metrics

Innovation

• Generative Spoken Language Modeling
• discrete speech representations
• bitrate efficiency
• speech resynthesis
• speech continuation