InsideSSL: Understanding Self-Supervised Speech Representations using a Model-Centric Perspective

2026-07-07

AI Summary

This work addresses the limited understanding of the dynamic evolution of internal representations in self-supervised speech models. We propose InsideSSL, a novel framework that integrates three intra-layer perspectives—information entropy, manifold curvature, and perturbation robustness—with an inter-layer generative compatibility matrix (GCM) to systematically characterize representational compression, geometric structure, and robustness from the model’s intrinsic viewpoint and link these properties to downstream performance. Our analysis reveals a stable phonetic core, fluctuating speaker information, and deep-layer semantic pruning within the models. Furthermore, we find that different pretraining objectives significantly influence acoustic compression and manifold unfolding, and that individual layers exhibit interpretable topological patterns in encoding phonemes, pitch, and speaker identity.

Abstract

Self-supervised learning (SSL) models, such as Wav2Vec2, HuBERT, and WavLM, have become foundational across a wide range of speech and audio tasks. Despite their success, understanding their internal layer-wise dynamics remains an ongoing challenge. To address this, we propose a two-part model-centric framework called InsideSSL. First, we establish a task-agnostic analysis from three intrinsic per-layer perspectives: compression (entropy), geometry (curvature), and robustness to perturbations. We show that varying training objectives induce distinct regimes of acoustic compression and manifold unfolding. Second, we introduce the cross-layer Generative Compatibility Matrix (GCM) to evaluate functional transferability, exposing stable phonetic cores, identity volatility, and deep-layer semantic pruning. In addition to these evaluations, linear probing connects the model-centric perspective to downstream tasks, demonstrating how layer topology dictates phoneme, pitch, and speaker encoding.

Problem

• self-supervised learning
• speech representations
• layer-wise dynamics
• model interpretability
• representation analysis

Innovation

• self-supervised speech representation
• model-centric analysis
• Generative Compatibility Matrix
• layer-wise dynamics
• intrinsic representation geometry