S-JEPA : Soft Clustering Anchors for Self-Supervised Speech Representation Learning

2026-06-17

AI Summary

This work proposes S-JEPA, a self-supervised speech representation learning framework based on the Joint-Embedding Predictive Architecture (JEPA), which overcomes limitations of traditional hard-clustering-based approaches that discard acoustic boundary ambiguity and require offline reclustering. S-JEPA aligns Gaussian Mixture Model (GMM) soft posterior probabilities at masked positions via KL divergence, enabling end-to-end continuous training while preserving acoustic uncertainty through a soft clustering objective. It eliminates the need for offline reclustering and introduces an unsupervised, signal-driven mechanism for adaptive input layer selection, removing reliance on manually specified clustering layers or teacher distillation. Evaluated on the SUPERB benchmark, S-JEPA achieves the lowest word error rate (WER) with under 90M parameters and matches HuBERT-Base performance in emotion recognition with roughly half the model size, demonstrating the efficacy of soft targets in self-supervised learning.

Abstract

Self-supervised speech encoders are predominantly trained by predicting discrete hard cluster IDs at masked positions, a recipe that collapses acoustic ambiguity at category boundaries and requires interrupting training to re-cluster the entire corpus between iterations. We introduce S-JEPA, a JEPA-style encoder-predictor pair trained to match the soft posteriors of a Gaussian Mixture Model at masked positions via KL divergence. Training runs as one continuous optimization trajectory in two phases: a fixed GMM over MFCC features, then an online GMM over encoder features, with the input layer selected adaptively from a label-free signal, removing both the offline re-cluster step and the hand-tuned choice of which transformer layer to cluster on. Under the SUPERB protocol, S-JEPA achieves the lowest WER among evaluated SSL methods below 90M parameters and matches HuBERT-Base on emotion recognition at roughly half its parameter count, establishing a new Pareto frontier without offline re-clustering or teacher distillation. An analysis of the predictor's per-frame entropy on held-out speech reveals a bimodal distribution with a substantial minority of frames near the entropy of a perfect two-cluster tie, providing direct empirical evidence that the soft-target objective preserves the acoustic ambiguity that hard targets would collapse. Code is available at https://github.com/gioannides/s-jepa.

Problem

• self-supervised learning
• speech representation
• hard clustering
• acoustic ambiguity
• re-clustering

Innovation

• soft clustering
• self-supervised learning
• Gaussian Mixture Model
• JEPA
• acoustic ambiguity