UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity MoE

2025-10-15

AI Summary

Auditory generation has long suffered from the artificial separation of speech and music modeling, leading to task conflicts and data imbalance that hinder the development of general-purpose audio synthesis models. To address this, we propose a dynamic-capacity Mixture-of-Experts (MoE) architecture that integrates domain-specific experts, shared experts, and skipable null experts, coupled with Top-P routing and a three-stage collaborative training strategy. This design enables cross-domain knowledge fusion and adaptive computational allocation. Our method effectively mitigates performance degradation during joint training, achieving state-of-the-art results on major speech (e.g., VALL-E X) and music (e.g., MusicGen, AudioLDM) benchmarks. It significantly enhances cross-domain cooperative learning capability while maintaining scalability and computational efficiency. The proposed framework establishes a unified, extensible, and high-performance paradigm for general audio generation.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: Generation
• Computer Vision: Diffusion Models for Vision

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent advances in unified multimodal models indicate a clear trend towards comprehensive content generation. However, the auditory domain remains a significant challenge, with music and speech often developed in isolation, hindering progress towards universal audio synthesis. This separation stems from inherent task conflicts and severe data imbalances, which impede the development of a truly unified audio generation model. To address this challenge, we propose UniMoE-Audio, a unified speech and music generation model within a novel Dynamic-Capacity Mixture-of-Experts (MoE) framework. Architecturally, UniMoE-Audio introduces a Top-P routing strategy for dynamic expert number allocation, and a hybrid expert design comprising routed experts for domain-specific knowledge, shared experts for domain-agnostic features, and null experts for adaptive computation skipping. To tackle data imbalance, we introduce a three-stage training curriculum: 1) Independent Specialist Training leverages original datasets to instill domain-specific knowledge into each "proto-expert" without interference; 2) MoE Integration and Warmup incorporates these specialists into the UniMoE-Audio architecture, warming up the gate module and shared expert using a subset of balanced dataset; and 3) Synergistic Joint Training trains the entire model end-to-end on the fully balanced dataset, fostering enhanced cross-domain synergy. Extensive experiments show that UniMoE-Audio not only achieves state-of-the-art performance on major speech and music generation benchmarks, but also demonstrates superior synergistic learning, mitigating the performance degradation typically seen in naive joint training. Our findings highlight the substantial potential of specialized MoE architecture and curated training strategies in advancing the field of universal audio generation. Homepage: https://mukioxun.github.io/Uni-MoE-site/home.html

Problem

• Unifying speech and music generation within a single audio synthesis model
• Addressing task conflicts and data imbalances in universal audio generation
• Developing dynamic-capacity MoE architecture for specialized audio domains

Innovation

• Dynamic-Capacity MoE framework for unified audio generation
• Top-P routing strategy with hybrid expert design
• Three-stage training curriculum addressing data imbalance