Neural Speaker Diarization via Multilingual Training: Evaluation on Low-Resource Nepali-Hindi Speech

2026-06-21

AI Summary

This work addresses the significant performance degradation in speaker diarization for low-resource languages such as Nepali and Hindi, primarily due to scarce annotated data. The study introduces DiaPer, the first application of the Perceiver architecture to this task, which leverages an attractor-based mechanism and is evaluated against EEND-EDA within a multilingual joint training framework. End-to-end training is conducted using LibriSpeech, VoxCeleb, and a newly collected NeHi corpus. On the NeHi test set, DiaPer achieves diarization error rates (DER) of 3.28%, 2.02%, 4.05%, and 4.76% for scenarios involving two, three, four speakers, and mixed conditions, respectively—substantially outperforming baseline models. The results demonstrate that DiaPer effectively mitigates language bias and enhances generalization in low-resource multilingual settings.

Abstract

Speaker diarization, the task of determining "who spoke when" in a multi-speaker recording, is a critical component in applications such as meeting transcription, accessibility tools, and multilingual information retrieval. While end-to-end neural diarization systems have achieved strong performance for English and other high-resource languages, their effectiveness degrades substantially for underrepresented languages where annotated speech data is scarce. This paper investigates speaker diarization for low-resource Nepali-Hindi speech through a multilingual training approach, comparing two modern architectures: EEND with encoder-decoder attractors (EEND-EDA) and EEND with Perceiver-based attractors (DiaPer). Both models are trained on a multilingual corpus combining English speech from LibriSpeech, diverse speaker recordings from VoxCeleb, and separately collected Nepali and Hindi audio, a setup designed to reduce language bias and encourage cross-lingual generalization. We evaluate both models across 2-speaker, 3-speaker, 4-speaker, and mixed-speaker scenarios on LibriSpeech, VoxCeleb, and Nepali-Hindi (NeHi) test sets. DiaPer achieves stronger overall performance than EEND-EDA, particularly in more challenging multi-speaker conditions, obtaining DERs of 3.28%, 2.02%, 4.05%, and 4.76% on NeHi 2-speaker, 3-speaker, 4-speaker, and mixed-speaker settings, respectively, compared to 1.50%, 9.68%, 16.17%, and 11.19% for EEND-EDA. These results demonstrate the viability of Perceiver-based end-to-end neural diarization for low-resource multilingual speech processing.

Problem

• speaker diarization
• low-resource languages
• multilingual speech
• Nepali-Hindi
• annotated speech data scarcity

Innovation

• multilingual training
• speaker diarization
• low-resource languages
• Perceiver-based attractors
• end-to-end neural diarization