Multimodal Speaker Identification in Classroom Environments

2026-06-10

AI Summary

This study addresses the challenge of poor performance in conventional acoustic speaker recognition within K–12 classrooms, where high variability in children’s speech and pervasive background noise degrade accuracy. To overcome this, the authors propose a novel multimodal speaker identification framework that integrates semantic context generated by a large language model (LLM) as an anchor with acoustic embeddings extracted by ECAPA-TDNN, followed by decision-level fusion via a gradient-boosting classifier. The approach substantially improves student identification accuracy in noisy conditions, raising overall performance from 39.0% to 50.3%, and achieving 76.9% accuracy (90.9% Top-3) for utterances longer than five seconds. Additionally, it attains 99.3% accuracy in distinguishing between teacher and student roles, offering robust support for automated instructional feedback systems.

Abstract

Automated analysis of K-12 classroom dynamics faces challenges due to background noise and variable child speech, often confounding acoustic-only models. This study evaluates a multimodal speaker identification framework anchoring acoustic embeddings with LLM-derived semantic context. Using a subset of the EDSI dataset (8 math classrooms, N = 2,801 utterances), we found an acoustic baseline (ECAPA-TDNN) achieved only 39.0% accuracy. By integrating transcript-based "contextual anchoring" into a gradient boosting classifier, our multimodal approach raised student identification to 50.3%. Performance also improved for utterances over 5 seconds, reaching 76.9% accuracy (vs. 64.9% baseline) with a 90.9% Top-3 accuracy. Additionally, the model distinguished teacher vs. student roles with 99.3% accuracy. This approach advances the feasibility of automated feedback systems capable of considering individual student participation, a crucial step for supporting equitable instruction at scale.

Problem

• Multimodal Speaker Identification
• Classroom Environments
• Acoustic-only Models
• Student Participation
• Automated Feedback Systems

Innovation

• multimodal speaker identification
• contextual anchoring
• LLM-derived semantics
• classroom audio analysis
• automated feedback systems