Stabilizing Short Duration Speaker Verification through Neural Re-scoring with Hybrid Enrollment

2026-06-14

AI Summary

Short-duration speaker verification suffers from degraded performance due to unstable speaker representations caused by limited speech duration, which renders the system highly susceptible to noise and phonetic variability. To address this challenge, this work constructs VoxPhrase, a large-scale short-utterance corpus derived from automatically segmented VoxCeleb data, and introduces a hybrid enrollment framework that integrates both text-dependent and text-independent paradigms within a neural rescoring architecture. The proposed approach uniquely leverages the complementary strengths of the two enrollment strategies and incorporates a frame-level parallel cross-attention mechanism to enable fine-grained comparison between test and enrollment utterances. Experimental results demonstrate consistent and significant performance gains across multiple state-of-the-art speaker embedding models, substantially improving robustness and accuracy in short-duration verification scenarios.

Abstract

Short-duration speaker verification (SDSV) is crucial for personalized keyword spotting, where test utterances are typically shorter than three seconds. Limited speech duration results in unstable speaker representations and increased sensitivity to noise and phoneme variations, thereby degrading performance. To investigate this issue, we construct VoxPhrase, a large-scale SDSV corpus automatically segmented from the VoxCeleb dataset. Our analysis shows that text-dependent (TD) enrollment is constrained by duration and yields unstable speaker representations. In contrast, although text-independent (TI) enrollment introduces content mismatch, its representations become more stable as the enrollment duration increases. Accordingly, we propose a hybrid-enrollment neural re-scoring framework that combines TD and TI enrollment and performs frame-level comparison via parallel cross-attention. Experiments on VoxPhrase demonstrate consistent improvements across multiple speaker models.

Problem

• short-duration speaker verification
• speaker representation stability
• text-dependent enrollment
• text-independent enrollment
• noise sensitivity

Innovation

• short-duration speaker verification
• hybrid enrollment
• neural re-scoring
• cross-attention
• VoxPhrase