Unified Audio Intelligence Without Regressing on Text Intelligence

2026-07-06

AI Summary

This work proposes a concise and unified audio-text large language model architecture that achieves state-of-the-art multimodal audio capabilities without compromising textual intelligence. Built upon a single Transformer decoder with a sparse mixture-of-experts (MoE) language model, the approach projects encoded audio into the text embedding space via an audio embedding projector, enabling unified processing of text tokens, quantized audio tokens, and projected audio embeddings. Through a combination of multi-stage supervised training, cascaded reinforcement learning, and multi-domain online distillation, the model attains state-of-the-art performance across diverse tasks—including automatic speech recognition, speech translation, text-to-speech synthesis, audio generation, and speech-to-speech translation—while fully preserving the base LLM’s strong capabilities in reasoning, alignment, factual knowledge, and long-context handling.

Abstract

Audio intelligence involves understanding, reasoning about, and generating both audio and speech. In this work, we introduce Nemotron-Labs-Audex-30B-A3B (Audex), a unified audio-text LLM built on Nemotron-Cascade-2-30B-A3B, a strong text-only MoE LLM. Audex adopts a simple unified design with a single Transformer decoder: audio inputs are encoded and projected into the text embedding space, while text tokens and quantized audio output tokens are treated uniformly during generation. This architecture enables strong audio-text fusion, seamless multimodal generation, and compatibility with standard LLM training and inference infrastructure. For training, we meticulously curate audio-text datasets comprising 157.4B audio tokens and 320.5B text tokens. We apply multi-stage supervised training on these datasets, followed by text-only Cascade RL and multi-domain on-policy distillation. Audex delivers state-of-the-art audio understanding, speech recognition and translation, text-to-speech, audio generation, and speech-to-speech generation, while preserving very compelling reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only LLM backbone with marginal or no regression. We release the model checkpoints to facilitate open research.

Problem

• audio intelligence
• unified audio-text LLM
• multimodal generation
• speech recognition
• text-to-speech

Innovation

• unified audio-text LLM
• single Transformer decoder
• audio-text fusion
• quantized audio tokens
• multimodal generation