PairAlign: A Framework for Sequence Tokenization via Self-Alignment with Applications to Audio Tokenization

2026-05-07

AI Summary

Existing audio tokenization methods struggle to simultaneously achieve sequence consistency, compactness, length control, proper end-of-sequence placement, and preservation of edit distance. This work formulates audio tokenization as a conditional sequence generation task and introduces a sequence-level self-alignment framework that optimizes global structure through dual-view mutual prediction. To effectively prevent many-to-one collapse, the approach incorporates several key mechanisms: vector quantization (VQ) initialization, exponential moving average (EMA) teacher targets, cross-pair teacher forcing, prefix corruption, likelihood contrastive learning, and explicit length control. Evaluated on 3-second speech segments, the method produces compact, non-degenerate symbolic sequences that, in TIMIT retrieval tasks, maintain competitive edit-distance performance using 55% fewer tokens.

Abstract

Many operations on sensory data -- comparison, memory, retrieval, and reasoning -- are naturally expressed over discrete symbolic structures. In language this interface is given by tokens; in audio, it must be learned. Existing audio tokenizers rely on quantization, clustering, or codec reconstruction, assigning tokens locally, so sequence consistency, compactness, length control, termination, and edit similarity are rarely optimized directly. We introduce PairAlign, a framework for compact audio tokenization through sequence-level self-alignment. PairAlign treats tokenization as conditional sequence generation: an encoder maps speech to a continuous condition, and an autoregressive decoder generates tokens from BOS, learning token identity, order, length, and EOS placement. Given two content-preserving views, each view's sequence is trained to be likely under the other's representation, while unrelated examples provide competing sequences. This gives a scalable surrogate for edit-distance preservation while discouraging many-to-one collapse. PairAlign starts from VQ-style tokenization and refines it with EMA-teacher targets, cross-paired teacher forcing, prefix corruption, likelihood contrast, and length control. On 3-second speech, PairAlign learns compact, non-degenerate sequences with broad vocabulary usage and strong cross-view consistency. On TIMIT retrieval, it preserves edit-distance search while reducing archive token count by 55%. A continuous-sweep probe shows lower local overlap than a dense geometric tokenizer, but stronger length control and bounded edit trajectories under 100 ms shifts. PairAlign is a sequence-symbolic predictive learner: like JEPA-style objectives, it predicts an abstract target from another view as a learned variable-length symbolic sequence, not a continuous latent.

Problem

• audio tokenization
• sequence consistency
• edit similarity
• length control
• discrete symbolic representation

Innovation

• self-alignment
• audio tokenization
• sequence generation
• edit-distance preservation
• symbolic representation