Exploring Token-Space Manipulation in Latent Audio Tokenizers

2026-05-11

AI Summary

This work proposes LATTE, a neural audio codec that addresses the limitations of existing frame-level sequential tokenization in modeling and manipulating global speech attributes—such as speaker identity and background noise—without task-specific supervision. By introducing learnable latent tokens preceding the audio features and quantizing only these non-temporally-aligned tokens for decoding, LATTE constructs a compact, discrete bottleneck representation. This design enables, for the first time, global editing in the audio token space through simple latent token swapping, facilitating unsupervised speaker conversion and denoising. Experiments demonstrate that LATTE achieves high-fidelity audio reconstruction at low bitrates while enabling efficient, controllable manipulation of global speech characteristics without any labeled data or auxiliary supervision.

Abstract

Neural audio codecs provide compact discrete representations for speech generation and manipulation. However, most codecs organize tokens as frame-level sequences, making it difficult to study or intervene on global factors of variation. In this work, we propose the Latent Audio Tokenizer for Token-space Editing (LATTE) that appends a fixed set of learnable latent tokens to the audio feature sequence and retains only these tokens for quantization and decoding. This design produces a compact, non-temporally aligned bottleneck in which each token can aggregate global information across the full utterance. We show that the resulting tokenizer preserves competitive reconstruction quality in low-bitrate speech coding settings while enabling simple token-space interventions. In particular, we find that swapping selected latent token positions between utterances can modify global attributes, such as speaker identity and background noise, and we evaluate these interventions on voice conversion and denoising tasks. Our results suggest that compact latent audio tokenizers can support controllable audio manipulation without supervision in task-specific editing models.

Problem

• audio tokenization
• global factors of variation
• token-space manipulation
• neural audio codecs
• controllable audio editing

Innovation

• latent audio tokenizer
• token-space manipulation
• global attribute editing
• neural audio codec
• unsupervised audio editing