Multi-Stage Speaker Diarization for Noisy Classrooms

2025-05-16

AI Summary

Addressing the severe degradation of speaker diarization (SD) performance in noisy classroom environments—characterized by low signal-to-noise ratios, overlapping child speech, and poor voice quality—this paper proposes a robust end-to-end SD framework. Our method integrates a hybrid speech activity detection (SAD) module that fuses self-supervised Transformer-based frame-level VAD with ASR-derived word-level timestamps, augmented by multi-stage denoising preprocessing and joint training on clean and noisy data. Implemented within the NVIDIA NeMo framework, the pipeline achieves 17% diarization error rate (DER) on teacher–student binary separation and 45% DER on full-speaker separation. Denoising substantially reduces false negatives (missed speech segments), while joint training significantly improves generalization to unseen noise conditions. The core contribution lies in the effective synergy between ASR-guided VAD modeling and noise-robust training paradigms, enabling reliable SD for challenging pediatric classroom scenarios.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Neuro-Symbolic Learning
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Speaker diarization, the process of identifying"who spoke when"in audio recordings, is essential for understanding classroom dynamics. However, classroom settings present distinct challenges, including poor recording quality, high levels of background noise, overlapping speech, and the difficulty of accurately capturing children's voices. This study investigates the effectiveness of multi-stage diarization models using Nvidia's NeMo diarization pipeline. We assess the impact of denoising on diarization accuracy and compare various voice activity detection (VAD) models, including self-supervised transformer-based frame-wise VAD models. We also explore a hybrid VAD approach that integrates Automatic Speech Recognition (ASR) word-level timestamps with frame-level VAD predictions. We conduct experiments using two datasets from English speaking classrooms to separate teacher vs. student speech and to separate all speakers. Our results show that denoising significantly improves the Diarization Error Rate (DER) by reducing the rate of missed speech. Additionally, training on both denoised and noisy datasets leads to substantial performance gains in noisy conditions. The hybrid VAD model leads to further improvements in speech detection, achieving a DER as low as 17% in teacher-student experiments and 45% in all-speaker experiments. However, we also identified trade-offs between voice activity detection and speaker confusion. Overall, our study highlights the effectiveness of multi-stage diarization models and integrating ASR-based information for enhancing speaker diarization in noisy classroom environments.

Problem

• Improving speaker diarization accuracy in noisy classrooms
• Evaluating denoising and hybrid VAD for speech detection
• Reducing Diarization Error Rate with multi-stage models

Innovation

• Multi-stage diarization using Nvidia's NeMo pipeline
• Hybrid VAD combining ASR and frame-level predictions
• Denoising improves diarization accuracy significantly