wav2tok 2.0: Scalable Audio Tokenization Maintaining Explicit Pairwise Token Alignment for Efficient Audio Retrieval

2026-06-25

AI Summary

This work addresses the challenge of preserving similarity and enabling efficient retrieval for variable-length speech segments in query-by-example spoken term detection (QbE-STD). The authors propose a staged training approach for audio tokenization that first learns discriminative, speaker-invariant representations through contrastive learning and vector quantization. Subsequently, they incorporate a CTC alignment loss and introduce an adaptive, frame-level prediction objective weighted by dynamic time warping (DTW) to achieve explicit pairwise token alignment. This strategy maintains alignment consistency while substantially improving model scalability. Empirical results demonstrate consistent superiority over both BEST-STD and general-purpose audio tokenizers on QbE-STD tasks, achieving a favorable balance between computational efficiency and strong generalization capability.

Abstract

Learning discrete speech representations that preserve similarity across variable-length utterances is central to query-by-example spoken term detection (QbE-STD). While wav2tok introduced CTC-based sequence alignment to enforce token consistency, its tightly coupled clustering and alignment training recipe limits scalability. We propose wav2tok 2.0, a scalable alignment-aware speech tokenizer built on the BEST-STD backbone. wav2tok 2.0 employs staged training, first learning discriminative, speaker-invariant representations via contrastive learning and vector quantization, and then enforcing pairwise token consistency using a CTC alignment loss and a novel DTW-aligned framewise prediction objective with adaptive weighting. Experiments show that wav2tok 2.0 consistently outperforms BEST-STD and general-purpose tokenizers on QbE-STD while remaining efficient and scalable.

Problem

• query-by-example spoken term detection
• discrete speech representations
• scalable audio tokenization
• pairwise token alignment
• variable-length utterances

Innovation

• alignment-aware tokenization
• staged training
• contrastive learning
• DTW-aligned framewise prediction
• scalable speech representation