PhiNet: Speaker Verification with Phonetic Interpretability

2026-04-02

AI Summary

This work addresses the limited transparency of automatic speaker verification systems, which hinders their deployment in high-accountability scenarios such as forensic voice comparison. Inspired by forensic experts’ reliance on phoneme-level evidence, we propose PhiNet—the first speaker verification network that incorporates phoneme-level interpretability. By integrating deep neural networks with phoneme alignment techniques, PhiNet achieves verification performance comparable to state-of-the-art black-box models on benchmark datasets including VoxCeleb, SITW, and LibriSpeech, while simultaneously offering both local and global interpretability of its decisions. This enables users to manually inspect phoneme-specific speaker characteristics and allows developers to perform targeted hyperparameter tuning and error analysis, thereby effectively bridging the gap between automatic speaker verification and forensic voice analysis practices.

Abstract

Despite remarkable progress, automatic speaker verification (ASV) systems typically lack the transparency required for high-accountability applications. Motivated by how human experts perform forensic speaker comparison (FSC), we propose a speaker verification network with phonetic interpretability, PhiNet, designed to enhance both local and global interpretability by leveraging phonetic evidence in decision-making. For users, PhiNet provides detailed phonetic-level comparisons that enable manual inspection of speaker-specific features and facilitate a more critical evaluation of verification outcomes. For developers, it offers explicit reasoning behind verification decisions, simplifying error tracing and informing hyperparameter selection. In our experiments, we demonstrate PhiNet's interpretability with practical examples, including its application in analyzing the impact of different hyperparameters. We conduct both qualitative and quantitative evaluations of the proposed interpretability methods and assess speaker verification performance across multiple benchmark datasets, including VoxCeleb, SITW, and LibriSpeech. Results show that PhiNet achieves performance comparable to traditional black-box ASV models while offering meaningful, interpretable explanations for its decisions, bridging the gap between ASV and forensic analysis.

Problem

• speaker verification
• interpretability
• phonetic evidence
• forensic speaker comparison
• transparency

Innovation

• phonetic interpretability
• speaker verification
• PhiNet
• explainable AI
• forensic speaker comparison