From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Training

2025-09-24

AI Summary

Existing end-to-end spoken dialogue models typically rely on multi-stage training and unified autoregressive modeling, overlooking the fundamental architectural disparities between text—characterized by strong inter-token causal dependencies—and speech audio—dominated by source-target relationships with weak temporal dependencies—leading to high computational overhead and inference latency. Method: We propose TtT, a unified Transformer framework initialized from a single large language model, jointly optimizing autoregressive text generation and non-autoregressive audio diffusion. Leveraging a source-driven, non-autoregressive joint training paradigm, TtT challenges the conventional assumption that audio tokens must be generated autoregressively. Contribution/Results: TtT achieves comparable or superior speech quality while significantly reducing training complexity and inference latency. Experiments demonstrate state-of-the-art performance in speech naturalness, interactive real-time capability, and training efficiency over multi-stage baselines.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Speech
• Computer Vision: Diffusion Models for Vision

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Recent advances in large language models have attracted significant interest in extending their capabilities to multimodal scenarios, particularly for speech-in speech-out conversational systems. However, existing multimodal models handling interleaved audio and text, such as MOSHI require complex multi stage training pipelines, incurring substantial computational costs. Moreover, these models uniformly apply autoregressive generation to both text and audio tokens, overlooking a fundamental asymmetry in their dependency structures: while text tokens exhibit strong target target dependencies requiring causal ordering, audio tokens are predominantly driven by source target dependencies, where audio outputs primarily condition on source text rather than preceding audio tokens. In this work, we propose TtT, a unified audio-text modeling framework that integrates AR text generation with non-autoregressive audio diffusion within a single Transformer architecture initialized from a pretrained LLM.

Problem

• Existing multimodal models require complex multi-stage training pipelines
• Autoregressive generation is inefficiently applied to both text and audio tokens
• Audio tokens lack strong sequential dependencies unlike text tokens

Innovation

• Non-autoregressive audio diffusion generation
• Joint training in single Transformer architecture
• Initialization from pretrained large language model