Unified Audio Generation and Editing via Joint Condition Modeling and Progressive Training

2026-06-15

AI Summary

Existing audio generation and editing methods often rely on task-specific architectures, resulting in systems that are complex and difficult to scale. This work proposes AudioWeave, the first unified framework that eliminates the need for task-dedicated modules by leveraging a diffusion Transformer architecture. Through joint conditional modeling, factorized positional encoding, and a multi-stage progressive training strategy, AudioWeave supports both text-to-audio generation and six distinct audio editing tasks within a single model. Experimental results demonstrate that AudioWeave achieves performance comparable to specialized models across all tasks, thereby validating the feasibility and potential of unified audio modeling.

Abstract

With the growing focus on audio in multimedia applications, numerous advanced works on audio generation have emerged. Existing studies typically treat text-to-audio (TTA) and other related audio generation tasks, such as instruction-based audio editing, as independent challenges, adopting task-specific architectures or modules. This absence of a unified modeling paradigm substantially increases the overhead and complexity of building a system for both audio generation and editing, while also leading to limited scalability. To address this issue, we introduce AudioWeave, a unified model for TTA and audio editing without additional task-specific components. Specifically, we propose a joint condition modeling approach with a factorized position embedding, enabling the diffusion transformer backbone to operate under heterogeneous inputs of TTA and audio editing. We further propose a progressive multistage training strategy to mitigate task competition and catastrophic forgetting caused by interference among multiple tasks. This in turn helps maintain the performance of each individual task and may even lead to improvements in certain aspects. Experimental results on TTA task and six audio editing tasks show that our unified model achieves competitive performance with task-specific models, laying a groundwork for further exploration of unified audio generation models.

Problem

• audio generation
• audio editing
• unified modeling
• text-to-audio
• task-specific architectures

Innovation

• unified audio generation
• joint condition modeling
• factorized position embedding
• progressive multistage training
• diffusion transformer