UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions

2026-04-24

AI Summary

This work addresses the longstanding fragmentation in generative audio modeling—traditionally split into isolated tasks such as text-to-speech (TTS), text-to-music (TTM), and text-to-audio (TTA)—which struggles to jointly handle structured semantic content and unstructured acoustic textures. To bridge this gap, we propose UniSonate, a unified generative framework based on flow matching that introduces a dynamic token injection mechanism to precisely map environmental sounds into a structured temporal latent space. Coupled with a multi-stage curriculum learning strategy, our approach harmonizes cross-modal optimization and, for the first time, enables accurate duration control of unstructured audio within a shared latent space. Joint training facilitates positive cross-modal transfer, significantly enhancing structural coherence and prosodic expressiveness. Experiments demonstrate that UniSonate outperforms task-specific baselines across instruction-driven TTS (WER: 1.47%), TTM (SongEval coherence: 3.18), and TTA tasks.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Generative audio modeling has largely been fragmented into specialized tasks, text-to-speech (TTS), text-to-music (TTM), and text-to-audio (TTA), each operating under heterogeneous control paradigms. Unifying these modalities remains a fundamental challenge due to the intrinsic dissonance between structured semantic representations (speech/music) and unstructured acoustic textures (sound effects). In this paper, we introduce UniSonate, a unified flow-matching framework capable of synthesizing speech, music, and sound effects through a standardized, reference-free natural language instruction interface. To reconcile structural disparities, we propose a novel dynamic token injection mechanism that projects unstructured environmental sounds into a structured temporal latent space, enabling precise duration control within a phoneme-driven Multimodal Diffusion Transformer (MM-DiT). Coupled with a multi-stage curriculum learning strategy, this approach effectively mitigates cross-modal optimization conflicts. Extensive experiments demonstrate that UniSonate achieves state-of-the-art performance in instruction-based TTS (WER 1.47%) and TTM (SongEval Coherence 3.18), while maintaining competitive fidelity in TTA. Crucially, we observe positive transfer, where joint training on diverse audio data significantly enhances structural coherence and prosodic expressiveness compared to single-task baselines. Audio samples are available at https://qiangchunyu.github.io/UniSonate/.

Problem

• text-to-speech
• text-to-music
• text-to-audio
• unified audio generation
• multimodal generation

Innovation

• unified audio generation
• dynamic token injection
• flow-matching
• multimodal diffusion transformer
• curriculum learning