Dasheng AudioGen: A Unified Model for Generating Coherent Audio Scenes from Text

2026-05-26

AI Summary

Existing audio generation models are often confined to a single sound type and struggle to synthesize coherent, mixed audio scenes comprising speech, music, and sound effects from text. This work proposes Dasheng AudioGen, a unified framework that enables fine-grained control over individual audio layers through structured, multi-perspective textual descriptions. By constructing a high-dimensional semantic-acoustic unified latent space, the model effectively disentangles and integrates concurrent sound components. Leveraging a flow-matching diffusion mechanism combined with a DiT architecture, Dasheng AudioGen supports end-to-end generation of high-quality mixed audio. Experiments demonstrate that the model achieves near-realistic quality in complex audio scene synthesis, matches specialized models on single-category tasks, and introduces a comprehensive evaluation benchmark for audio scene generation.

Abstract

Audio generation has long been fragmented, with speech, music, and sound effects produced by domain-specific models that fail to jointly generate coherent audio scenes from a single description. The key obstacles are insufficient fine-grained supervision for real-world mixed audio and limited acoustic representations for modeling concurrent audio components. We present Dasheng AudioGen, a unified framework for generating general mixed-audio scenes from text. Dasheng AudioGen introduces structured multi-view captions, which explicitly decouple complex acoustic scenes into complementary description views, thereby enabling fine-grained control over audio layers. Furthermore, we employ a high-dimensional unified semantic-acoustic representation as the shared latent space. It injects semantic priors that facilitate cross-modal training convergence, while its high-dimensional feature space provides sufficient capacity to disentangle and fuse concurrent audio components effectively. With these designs, a simple flow-matching DiT achieves high-quality end-to-end audio scene generation. We also establish a comprehensive evaluation pipeline for audio scene generation. Experiments demonstrate that Dasheng AudioGen achieves performance approaching real-world recordings in mixed-audio categories, while remaining competitive with specialized models in single-type generation tasks. Demos are available at https://nieeim.github.io/Dasheng-AudioGen-Web/.

Problem

• audio generation
• coherent audio scenes
• mixed-audio
• text-to-audio
• acoustic representation

Innovation

• structured multi-view captions
• unified semantic-acoustic representation
• flow-matching DiT
• coherent audio scene generation
• fine-grained audio control