LISE : Listenable Interpretable Speaker Embeddings

2026-06-19

AI Summary

This work addresses the lack of interpretability in existing deep speaker embeddings, which hinders verification of whether they encode speech characteristics aligned with human perception. The authors propose an unsupervised framework that decomposes pretrained embeddings—such as x-vectors and ECAPA-TDNN outputs—into a small set of perceptually discriminable, structured components, without requiring any attribute annotations. This approach achieves, for the first time, audibly interpretable speaker representations. Remarkably, it incurs negligible degradation in automatic speaker verification (ASV) performance—evidenced by an almost unchanged equal error rate (EER)—while enabling human listeners to identify speakers from the reconstructed components with 83.9% accuracy in listening experiments. This breakthrough facilitates direct human analysis and validation of the content encoded within speaker embeddings.

Abstract

Deep neural network-based automatic speaker verification (ASV) systems achieve impressive performance but their embedding representations remain opaque, lacking a structured and perceptually verifiable explanation of the vocal characteristics they encode. Existing approaches either require annotation of speaker attributes or introduce alternative representations whose interpretability is unvalidated with listeners. We propose Listenable Interpretable Speaker Embeddings (LISE), a label-free framework that decomposes pretrained speaker embeddings into a small set of components. This decomposition yields a structured representation that supports the analysis of what information has been encoded by speaker embeddings. LISE preserves ASV performance with negligible EER degradation on x-vector and ECAPA-TDNN. Crucially, the interpretability of these components for human listeners is demonstrated through listening experiments, where participants distinguished speakers with 83.9% accuracy.

Problem

• speaker verification
• interpretable embeddings
• perceptual validation
• speaker characteristics
• embedding transparency

Innovation

• interpretable speaker embeddings
• label-free decomposition
• listenability
• speaker verification
• perceptual evaluation