Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing

2026-04-12

AI Summary

Existing approaches lack a unified end-to-end framework capable of jointly handling audio understanding, generation, and editing across diverse audio domains such as sound, music, and speech. This work proposes the first cross-audio-domain unified end-to-end system, integrating the high-level reasoning capabilities of multimodal large language models with the high-fidelity synthesis power of trainable diffusion Transformers. To support this framework, the authors introduce AudioEdit, a million-scale audio editing dataset. The proposed system enables knowledge-enhanced reasoning, context-aware generation, and zero-shot cross-lingual control, achieving performance on par with or surpassing that of specialized expert models across multiple benchmarks, thereby demonstrating remarkable versatility and state-of-the-art generative capabilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation
• Computer Vision: Diffusion Models for Vision

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Recent progress in multimodal models has spurred rapid advances in audio understanding, generation, and editing. However, these capabilities are typically addressed by specialized models, leaving the development of a truly unified framework that can seamlessly integrate all three tasks underexplored. While some pioneering works have explored unifying audio understanding and generation, they often remain confined to specific domains. To address this, we introduce Audio-Omni, the first end-to-end framework to unify generation and editing across general sound, music, and speech domains, with integrated multi-modal understanding capabilities. Our architecture synergizes a frozen Multimodal Large Language Model for high-level reasoning with a trainable Diffusion Transformer for high-fidelity synthesis. To overcome the critical data scarcity in audio editing, we construct AudioEdit, a new large-scale dataset comprising over one million meticulously curated editing pairs. Extensive experiments demonstrate that Audio-Omni achieves state-of-the-art performance across a suite of benchmarks, outperforming prior unified approaches while achieving performance on par with or superior to specialized expert models. Beyond its core capabilities, Audio-Omni exhibits remarkable inherited capabilities, including knowledge-augmented reasoning generation, in-context generation, and zero-shot cross-lingual control for audio generation, highlighting a promising direction toward universal generative audio intelligence. The code, model, and dataset will be publicly released on https://zeyuet.github.io/Audio-Omni.

Problem

• audio generation
• audio editing
• multimodal understanding
• unified framework
• general audio domains

Innovation

• unified audio framework
• multimodal understanding
• diffusion transformer
• audio editing dataset
• zero-shot cross-lingual control