MOSS-Audio-Tokenizer: Scaling Audio Tokenizers for Future Audio Foundation Models

2026-02-11

AI Summary

Existing discrete audio tokenizers rely on pretrained encoders, semantic distillation, or heterogeneous CNN architectures, introducing fixed inductive biases that constrain reconstruction fidelity and scalability. This work proposes CAT—a fully end-to-end, homogeneous, and scalable audio tokenization method—that, for the first time, employs a purely causal Transformer to jointly optimize the encoder, quantizer, and decoder without auxiliary modules. Leveraging a 1.6-billion-parameter model pretrained on 3 million hours of diverse audio, CAT consistently outperforms existing codecs across speech, sound, and music at various bitrates. The resulting discrete tokens enable the first purely autoregressive text-to-speech system that surpasses non-autoregressive counterparts and demonstrate competitive performance in automatic speech recognition tasks.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Computer Vision: Diffusion Models for Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Discrete audio tokenizers are fundamental to empowering large language models with native audio processing and generation capabilities. Despite recent progress, existing approaches often rely on pretrained encoders, semantic distillation, or heterogeneous CNN-based architectures. These designs introduce fixed inductive biases that limit reconstruction fidelity and hinder effective scaling. In this paper, we argue that discrete audio tokenization should be learned fully end-to-end using a homogeneous and scalable architecture. To this end, we first propose CAT (Causal Audio Tokenizer with Transformer), a purely Transformer-based architecture that jointly optimizes the encoder, quantizer, and decoder from scratch for high-fidelity reconstruction. Building on the CAT architecture, we develop MOSS-Audio-Tokenizer, a large-scale audio tokenizer featuring 1.6 billion parameters, pre-trained on 3 million hours of diverse, general audio data. We show that this simple, fully end-to-end approach built from homogeneous, causal Transformer blocks scales gracefully and supports high-fidelity reconstruction across diverse audio domains. Across speech, sound, and music, MOSS-Audio-Tokenizer consistently outperforms prior codecs over a wide range of bitrates, while exhibiting predictable improvements with increased scale. Notably, leveraging the discrete tokens from our model, we develop the first purely autoregressive TTS model that surpasses prior non-autoregressive and cascaded systems. Furthermore, MOSS-Audio-Tokenizer enables competitive ASR performance without auxiliary encoders. Our findings position the CAT architecture as a unified, scalable interface for the next generation of native audio foundation models.

Problem

• audio tokenizer
• discrete representation
• scalability
• inductive bias
• audio foundation models

Innovation

• end-to-end audio tokenization
• homogeneous Transformer architecture
• scalable audio foundation model
• causal Transformer
• discrete audio representation