SonicWeave: Chunk-Routed Mixture-of-Experts for Unified Audio Scene Generation

2026-08-10

AI Summary

This work addresses the structural conflicts arising from heterogeneous audio components—speech, music, and sound effects—in unified audio generation, which challenge the use of a shared backbone and demand fine-grained adaptability within individual segments. To this end, the authors propose SonicWeave, a flow-matching-based unified generative model featuring a Chunk-wise Prior-Evidence Mixture-of-Experts (CPE-MoE) mechanism. Its core innovation is a conflict-gated prior-evidence routing strategy that dynamically balances global textual conditioning and local contextual cues based on the reliability of local acoustic states during the diffusion process, enabling temporally coherent expert dispatching. Experiments demonstrate that SonicWeave outperforms both dense and vanilla MoE baselines across TTS, TTA, and TTM benchmarks, with superior performance in complex compositional scenarios; routing analysis further reveals content-dependent specialization of experts across diffusion stages.

Abstract

Text-conditioned general audio generation is moving beyond isolated speech, music, and sound-effect synthesis toward a single model that can compose them into controllable, coherent audio scenes. This unified setting is particularly challenging: heterogeneous components impose conflicting structural requirements on a shared backbone, while a complex mixed scene may contain locally distinct or overlapping content that demands fine-grained adaptation within the same clip. Existing audio mixture-of-experts (MoEs) mainly route at the domain level, while token-wise routing overlooks the local continuity inherent to acoustic signals. We propose SonicWeave, a flow-matching model for unified audio scene generation. At its core is a chunk-routed MoE with a conflict-gated prior-evidence routing mechanism (CPE-MoE). CPE-MoE routes contiguous acoustic chunks by combining a global prior that encodes the structured text condition and diffusion phase with local evidence from the evolving acoustic state. A learned conflict gate favors the prior when local states are unreliable, while allowing local evidence to influence routing when a region departs from the global scene context. SonicWeave supports speech, music, sound effects, singing, and their fine-grained mixtures with a single set of weights. Across TTS, TTA, and TTM benchmarks, SonicWeave consistently improves over controlled Dense and Base-MoE baselines. Complex-scene evaluation further demonstrates improved compositional quality, while routing analyses reveal content-dependent expert specialization across diffusion phases. These results suggest that temporally coherent, prior-evidence routing is an effective conditional-computation strategy for unified audio generation. Project page: https://caiyunrui.github.io/SonicWeave.

Problem

• unified audio generation
• audio scene composition
• mixture-of-experts
• fine-grained adaptation
• heterogeneous audio components

Innovation

• chunk-routed MoE
• conflict-gated routing
• unified audio generation
• flow-matching
• prior-evidence routing