Semi-Supervised Speech Confidence Detection using Pseudo-Labelling and Whisper Embeddings

2026-06-15

AI Summary

This study addresses the lack of effective methods for automatically assessing speaker confidence in educational settings. The authors propose a multimodal modeling framework that integrates embeddings from the Whisper encoder with handcrafted acoustic features—such as pitch, speech rate, and disfluency measures—thereby introducing pretrained speech representations to the task of confidence detection for the first time. To overcome limited labeled data, they employ a pseudo-labeling strategy and design a co-attention mechanism to enable efficient fusion of heterogeneous features. The proposed approach achieves 75% accuracy in speaker confidence detection, offering a viable technical pathway for delivering personalized spoken feedback and adaptive learning interventions.

Abstract

Understanding speaker confidence is crucial in educational settings, as it can enhance personalised feedback and improve learning outcomes. This study introduces a novel framework for detecting speaker confidence by integrating human-engineered features with embeddings from the Whisper encoder. To address data limitations, a pseudo-labelling technique is employed to expand the labelled dataset, allowing the model to learn from both human-annotated and model-generated labels. The framework combines traditional speech features including pitch, volume, rate of speech, and the presence of disfluencies and stress, with Whisper embeddings, and uses a co-attention mechanism to fuse these representations and achieve an overall accuracy of 75%. This study contributes to advancing speech analysis, enabling applications that support personalised learning and speaking skill development.

Problem

• speech confidence detection
• semi-supervised learning
• educational technology
• speaker confidence
• personalised feedback

Innovation

• pseudo-labelling
• Whisper embeddings
• co-attention mechanism
• semi-supervised learning
• speech confidence detection