MARS: Audio Generation via Multi-Channel Autoregression on Spectrograms

2025-09-30

AI Summary

This work addresses the challenges of inadequate harmonic structure modeling and temporal coherence in high-fidelity audio generation. We propose a multi-channel spectrogram autoregressive modeling framework. Our key contributions are: (1) treating the spectrogram as a multi-channel image and introducing Channel Multiplexing (CMX) to compress spatial dimensions while preserving full-band spectral information; (2) designing a cross-scale, non-token-level progressive refinement mechanism to avoid distortion inherent in conventional tokenization-based discretization; and (3) employing a shared tokenizer to ensure consistency across multi-scale discrete representations and leveraging a Transformer-based architecture for efficient, scalable autoregression. Experiments on large-scale datasets demonstrate state-of-the-art (SOTA) or superior performance in STFT reconstruction fidelity, speech MOS, and music Fréchet Audio Distance (FAD), significantly improving audio fidelity and fine-grained perceptual detail.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Research on audio generation has progressively shifted from waveform-based approaches to spectrogram-based methods, which more naturally capture harmonic and temporal structures. At the same time, advances in image synthesis have shown that autoregression across scales, rather than tokens, improves coherence and detail. Building on these ideas, we introduce MARS (Multi-channel AutoRegression on Spectrograms), a framework that treats spectrograms as multi-channel images and employs channel multiplexing (CMX), a reshaping technique that lowers height and width without discarding information. A shared tokenizer provides consistent discrete representations across scales, enabling a transformer-based autoregressor to refine spectrograms from coarse to fine resolutions efficiently. Experiments on a large-scale dataset demonstrate that MARS performs comparably or better than state-of-the-art baselines across multiple evaluation metrics, establishing an efficient and scalable paradigm for high-fidelity audio generation.

Problem

• Developing spectrogram-based autoregressive framework for audio generation
• Enhancing coherence and detail through multi-scale channel multiplexing
• Establishing efficient scalable paradigm for high-fidelity audio synthesis

Innovation

• Multichannel autoregression on spectrogram images
• Channel multiplexing reshaping for efficient processing
• Shared tokenizer enables cross-scale coherent refinement