CPT-Boosted Wav2vec2.0: Towards Noise Robust Speech Recognition for Classroom Environments

2024-09-13

arXiv.org

AI Summary

In classroom environments, automatic speech recognition (ASR) performance degrades significantly due to multiple concurrent noise sources (e.g., overlapping student speech, HVAC/projector noise), multi-microphone acquisition, and strong reverberation. To address this, we propose a domain adaptation method for Wav2vec 2.0 based on continual pretraining (CPT), the first systematic validation of CPT on real-world classroom speech. Our approach performs self-supervised representation refinement using only unlabeled classroom recordings—requiring no downstream fine-tuning—thereby enhancing cross-noise-type, cross-device, and cross-environment generalization robustness. Experiments demonstrate a relative word error rate (WER) reduction exceeding 10%, with particularly substantial improvements under low signal-to-noise ratio (SNR), high reverberation, and far-field recording conditions. This work establishes a novel paradigm for lightweight, adaptation-free robust ASR in educational AI applications.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: Speech
• Application Domains: Humanities & Computational Social Science

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Creating Automatic Speech Recognition (ASR) systems that are robust and resilient to classroom conditions is paramount to the development of AI tools to aid teachers and students. In this work, we study the efficacy of continued pretraining (CPT) in adapting Wav2vec2.0 to the classroom domain. We show that CPT is a powerful tool in that regard and reduces the Word Error Rate (WER) of Wav2vec2.0-based models by upwards of 10%. More specifically, CPT improves the model's robustness to different noises, microphones and classroom conditions.

Problem

• Develop robust ASR systems for classroom environments
• Adapt Wav2vec2.0 using continued pretraining (CPT)
• Reduce Word Error Rate (WER) in noisy classroom conditions

Innovation

• Continued pretraining enhances Wav2vec2.0 robustness.
• CPT reduces Word Error Rate significantly.
• Improved ASR performance in noisy classrooms.