AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation

2026-06-22

AI Summary

This work addresses the challenge of unified generation of speech, sound effects, and music—a task requiring high fidelity, end-to-end training, effective contextual modeling, and variable-length synthesis, which existing approaches struggle to balance. The authors propose AudioCALM, a framework that extends autoregressive language modeling to continuous audio latent spaces by integrating flow matching (predicting rectified-flow velocities) with a block-causal AR-Flow attention mechanism, enabling high-quality generation of arbitrary duration. A novel asymmetric multimodal mixture-of-experts architecture (A-MoME) and a unified conditioning interface are introduced: speech utilizes dedicated residual experts, while sound effects and music share a common backbone, incurring no additional inference cost. Experiments demonstrate that AudioCALM matches or surpasses specialized models across all three modalities and significantly outperforms current unified audio generation methods.

Abstract

Unifying speech, sound, and music generation in one model is hindered by tradeoffs between fidelity, end-to-end training, in-context conditioning, and variable-length synthesis that no current paradigm fully resolves. To address this challenge, we present AudioCALM, a universal audio generation framework that extends autoregressive (AR) next-token prediction from discrete tokens to continuous audio latents: a thin flow-matching head replaces the softmax to predict rectified-flow velocities at each position, and a block-causal AR-Flow attention pattern produces arbitrary-length output. Joint training of multiple audio generation tasks faces an asymmetric text--audio mismatch: speech transcripts align to specific time spans and demand tight, time-aligned attention, whereas sound and music captions describe only overall semantics and rely on diffuse, holistic attention; mixing the two disproportionately degrades sound and music generation. We address this asymmetry at two levels: a data reformulation strategy that unifies all three tasks under a single description-style conditioning interface, and a novel architecture Asymmetric Mixture-of-Modality-Experts (A-MoME), which adds a dedicated residual expert for speech while sound and music share the backbone, incurring no inference overhead on non-speech inputs. Experimental results demonstrate that AudioCALM matches modality-specific state-of-the-art and outperforms prior unified baselines on speech, sound, and music generation benchmarks.

Problem

• universal audio generation
• autoregressive modeling
• text-audio alignment
• variable-length synthesis
• multimodal asymmetry

Innovation

• continuous autoregressive modeling
• flow matching
• asymmetric modality alignment
• universal audio generation
• Mixture-of-Experts