LoSATok: Low-dimensional Semantic-Acoustic Tokenizer for Cross-Domain Audio Understanding and Generation

2026-05-26

AI Summary

This work addresses the challenge that existing unified audio tokenizers entangle semantic and acoustic information within high-dimensional continuous latent spaces, thereby increasing the burden on generative models. To mitigate this, the authors propose LoSATok—a low-dimensional semantic-acoustic tokenizer—that compresses 1280-dimensional semantic features into a compact 128-dimensional representation via a semantic bottleneck. A temporal relational loss is introduced to preserve sequential structure, and a dual-level semantic supervision mechanism is designed to jointly optimize comprehension and generation capabilities within the compressed latent space. Integrated with a Diffusion Transformer (DiT) architecture, LoSATok achieves substantial improvements in generation quality across speech, music, and general audio tasks while maintaining strong performance in understanding benchmarks.

Abstract

Audio tokenizers are fundamental to unifying audio understanding and generation. Understanding requires high-level semantics, while generation demands semantic and acoustic details. Existing unified tokenizers jointly encode both in high-dimensional continuous latents, which increases the modeling burden of Diffusion Transformers (DiTs) for generation. We propose LoSATok, a low-dimensional audio tokenizer for cross-domain audio understanding and generation. Motivated by the observation that 1280-dimensional semantic encoder features are compressible, we introduce a Semantic Bottleneck that compresses them into 128 dimensions, regularized by the proposed time-relation loss for temporal feature consistency. We further design a dual-level semantic supervision method that leverages both high- and low-dimensional semantic signals, enabling the tokenizer to jointly capture semantics and acoustic details within a compact latent space. Experiments on speech, music, and general audio show that SemBo preserves strong low-dimensional semantic capacity and LoSATok retains competitive understanding performance compared with several semantic representations, while consistently improving DiT modeling performance on speech, music, and audio generation. These results demonstrate that LoSATok's low-dimensional representations can effectively support audio understanding and generation. Our code is provided at https://github.com/wxzyd123/LoSATok.

Problem

• audio tokenizer
• cross-domain audio
• semantic-acoustic representation
• low-dimensional representation
• Diffusion Transformers

Innovation

• low-dimensional tokenizer
• semantic bottleneck
• dual-level semantic supervision
• time-relation loss
• cross-domain audio generation