Speaker Identity in Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts Approach

2026-06-19

AI Summary

This study addresses the limited generalization of existing speaker verification systems to non-linguistic vocalizations (NVVs) and the catastrophic forgetting of speech-based tasks during fine-tuning. It presents the first systematic investigation into speaker consistency across ten categories of NVVs and proposes a joint optimization framework that integrates frozen Data2Vec features with ECAPA-TDNN. The approach incorporates a mixture-of-experts module with domain-aware routing and leverages conditional knowledge distillation alongside contrastive learning to effectively reduce the domain gap between speech and NVVs. The method not only enhances NVV verification performance but also improves the original speech-based verification: the cross-domain equal error rate (EER) between speech and NVVs drops from 38.93% to 22.66%, while the speech-only EER improves from 13.17% to 9.24%.

Abstract

As expressive text-to-speech (TTS) and voice conversion (VC) systems increasingly generate non-verbal vocalizations (NVVs) to enhance naturalness, reliable speaker verification (SV) becomes essential to objectively assess identity consistency across both verbal and non-verbal segments. Yet current SV systems generalize poorly to NVVs, and fine-tuning on NVV data causes catastrophic forgetting of speech performance. We present the first systematic study across 10 NVV types and propose a framework combining frozen Data2Vec self-supervised features with ECAPA-TDNN, enhanced by a Mixture of Experts (MoE) module with learned domain-aware routing. A conditional distillation loss on speech inputs via a pretrained teacher retains speech-to-speech accuracy, while a contrastive loss bridges the speech-NVV domain gap. Our method reduces speech-NVV EER from 38.93% to 22.66% over a pretrained baseline, and improves speech EER from 13.17% to 9.24% via distillation.

Problem

• speaker verification
• non-verbal vocalizations
• catastrophic forgetting
• identity consistency
• domain generalization

Innovation

• Mixture of Experts
• Conditional Distillation
• Non-Verbal Vocalizations
• Speaker Verification
• Domain Gap