UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models

2025-10-06

AI Summary

Current ASR and TTS systems are typically modeled separately; while discrete speech tokenization enables joint modeling, it suffers from information loss that limits performance. This paper introduces the first unified large language model framework based on continuous speech representations for end-to-end joint ASR and TTS modeling. Our approach addresses the fundamental trade-off between recognition and generation through two core innovations: (1) a dual-attention mechanism that dynamically switches between causal and bidirectional masking to jointly optimize autoregressive speech recognition and flow-matching-based speech synthesis; and (2) a text-prefixed speech completion strategy enabling high-fidelity zero-shot voice cloning. Experiments demonstrate that our framework matches or surpasses state-of-the-art single-task models on both ASR and zero-shot text-to-speech benchmarks—marking the first empirical validation of the effectiveness and superiority of continuous-representation-based unified speech understanding and generation.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Large language models (LLMs) have demonstrated promising performance in both automatic speech recognition (ASR) and text-to-speech (TTS) systems, gradually becoming the mainstream approach. However, most current approaches address these tasks separately rather than through a unified framework. This work aims to integrate these two tasks into one unified model. Although discrete speech tokenization enables joint modeling, its inherent information loss limits performance in both recognition and generation. In this work, we present UniVoice, a unified LLM framework through continuous representations that seamlessly integrates speech recognition and synthesis within a single model. Our approach combines the strengths of autoregressive modeling for speech recognition with flow matching for high-quality generation. To mitigate the inherent divergence between autoregressive and flow-matching models, we further design a dual attention mechanism, which switches between a causal mask for recognition and a bidirectional attention mask for synthesis. Furthermore, the proposed text-prefix-conditioned speech infilling method enables high-fidelity zero-shot voice cloning. Experimental results demonstrate that our method can achieve or exceed current single-task modeling methods in both ASR and zero-shot TTS tasks. This work explores new possibilities for end-to-end speech understanding and generation.

Problem

• Unifying speech recognition and synthesis in one model
• Overcoming information loss from discrete speech tokenization
• Integrating autoregressive and flow-matching models for speech tasks

Innovation

• Unified LLM framework with continuous speech representations
• Combines autoregressive ASR and flow-matching TTS
• Dual attention mechanism switches between recognition and synthesis