Scaling Properties of Continuous Diffusion Spoken Language Models

2026-04-27

AI Summary

This work addresses the longstanding performance gap between speech-specific language models and their text or multimodal counterparts, which has been attributed to computational and data bottlenecks inherent in discrete autoregressive architectures. The authors propose a scalable continuous diffusion-based speech language model, trained on an unprecedented scale—up to 16 billion parameters and tens of millions of hours of conversational speech—and introduce a novel phoneme-level Jensen-Shannon divergence (pJSD) metric for evaluation. Their findings reveal that continuous diffusion models obey scaling laws, exhibiting loss insensitivity to specific choices of data and model size, and demonstrate that the optimal token-to-parameter ratio decreases with increasing compute, favoring efficient inference. The model generates expressive, prosodically rich, multi-speaker, and multilingual speech, though challenges remain in maintaining long-form coherence.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Speech-only spoken language models (SLMs) lag behind text and text-speech models in performance, with recent discrete autoregressive (AR) SLMs indicating significant computational and data demands to match text models. Since discretizing continuous speech for AR creates bottlenecks, we explore whether continuous diffusion (CD) SLM is more viable. To quantify the SLMs linguistic quality, we introduce the phoneme Jensen-Shannon divergence (pJSD) metric. Our analysis reveals CD SLMs, mirroring AR behavior, exhibit scaling laws for validation loss and pJSD, and show optimal token-to-parameter ratios decreasing as compute scales. However, for the latter, loss becomes insensitive to choice of data and model sizes, showing potential for fast inference. Scaling CD SLMs to 16B parameters with tens of millions of hours of conversational data enables generation of emotive, prosodic, multi-speaker, multilingual speech, though achieving long-form coherence remains a significant challenge.

Problem

• continuous diffusion
• spoken language models
• long-form coherence
• scaling laws
• speech generation

Innovation

• continuous diffusion
• spoken language models
• scaling laws
• phoneme Jensen-Shannon divergence
• speech generation