Speaker-Disentangled Chunk-Wise Regression for Syllabic Tokenization

2026-07-04

AI Summary

This work addresses the challenge that unsupervised syllable segmentation is often confounded by speaker identity, leading to learned representations that entangle linguistic content with non-linguistic speaker-specific information. To explicitly disentangle these factors, the authors propose a chunk-based teacher–student distillation framework: leveraging a pretrained HuBERT model as the teacher, they generate student representations within fixed-length speech chunks using a speaker perturbation strategy and align them to clean teacher targets via a chunk-level regression loss. This approach significantly enhances the purity and generalizability of syllable tokens, achieving state-of-the-art performance on syllable boundary detection and clustering tasks. Furthermore, language models trained on the resulting syllables outperform phoneme-level SpiRit-LM by 7% on syntactic and semantic understanding benchmarks.

Abstract

Unsupervised syllabic tokenization aims to learn discrete syllabic tokens that capture latent linguistic content-related structure from raw speech. Recent syllabic tokenization methods employ teacher-student distillation of the pretrained HuBERT to organize latent speech frame representations into syllabic segments. However, when trained with an utterance-level cross-entropy objective, the model predicts speaker identity rather than linguistic content, thereby compromising the purity of syllabic tokens. To address this problem, we propose a speaker-disentangled syllabic tokenizer that regresses speaker-perturbed student representations toward clean teacher targets within fixed-length chunks. Experimental results demonstrate that our proposed method achieves state-of-the-art performance in syllable boundary detection and syllabic segment clustering. Moreover, a speech language model trained on our syllabic tokens achieves a 7% relative improvement in syntactic and semantic understanding over the phone-level SpiRit-LM.

Problem

• unsupervised syllabic tokenization
• speaker identity
• linguistic content
• syllabic tokens
• cross-entropy objective

Innovation

• speaker-disentangled
• chunk-wise regression
• syllabic tokenization
• unsupervised speech representation
• teacher-student distillation