Disentangling Speaker and Language Effects in Cross-Lingual Speaker Verification for Iberian Languages

2026-07-01

AI Summary

This work addresses the significant performance degradation in cross-lingual speaker verification caused by language mismatch, a challenge exacerbated by existing evaluation protocols that conflate linguistic and speaker-specific effects. To disentangle these factors, the authors construct a bilingual same-speaker evaluation set spanning five Iberian languages, enabling—for the first time—the isolation of language and speaker effects under fixed speaker identity. Building upon the HuBERT model, they introduce a Cross-Lingual Transfer Matrix (CLTM) to quantitatively assess the respective contributions of language and speaker variability to verification performance. Experimental results demonstrate that language mismatch is the dominant source of performance loss, whereas speaker-related variation has a comparatively limited impact, thereby offering a clear direction for optimizing cross-lingual speaker verification systems.

Abstract

Cross-lingual speaker verification (SV) systems typically exhibit performance degradation when enrollment and test utterances are spoken in different languages. However, standard evaluation protocols confound language mismatch with inter-speaker variability, as evaluation is generally performed with different speakers across languages. In this work, we introduce a bilingual same-speaker evaluation set for five Iberian languages, enabling analysis of cross-lingual SV under constant speaker identity. We apply this setup to a HuBERT-based SV system previously shown to exhibit strong language dependence, and analyze results using the Cross-Lingual Transfer Matrix (CLTM) to study pairwise cross-lingual transfer. Our results show that speaker-related variability accounts for part of the observed degradation, but language mismatch remains the main driver of cross-lingual performance loss. These findings provide a more precise characterization of language dependence in cross-lingual SV.

Problem

• cross-lingual speaker verification
• language mismatch
• speaker variability
• Iberian languages
• language dependence

Innovation

• cross-lingual speaker verification
• bilingual same-speaker evaluation
• language disentanglement
• HuBERT-based SV
• Cross-Lingual Transfer Matrix