Enhancing BEST-RQ Pseudo-Label Quality through Online Refinement for Automatic Speech Recognition

2026-06-24

AI Summary

This work addresses the limitation of BEST-RQ’s fixed online quantization mechanism, which constrains pseudo-label quality and yields weak supervision signals in self-supervised speech representation learning. While preserving the model’s architectural simplicity, the authors propose three key enhancements: replacing the linear projection with principal component analysis (PCA), iterative codebook optimization, and distillation-assisted codebook updates. These modifications substantially improve pseudo-label fidelity. Experimental results on the LibriSpeech dataset demonstrate that the proposed approach reduces the word error rate on the test-other subset from 10.1% to 8.8%, achieving a relative improvement of 12%.

Abstract

BEST-RQ is a simple and effective self-supervised training method for speech representation learning that performs well on automatic speech recognition (ASR) tasks. It generates pseudolabels using a fixed online quantization scheme, which simplifies training but provides weaker supervision than HuBERT-style models that iteratively refine pseudo-labels. In this work, we improve online pseudo-label generation while preserving simplicity. We propose three modifications: replacing the quantizer's linear projection with Principal Component Analysis (PCA), updating the codebook via iterative codebook refinement, and introducing an additional codebook updated via codebook distillation. We pre-train on the LibriSpeech 960-hour dataset and fine-tune using 100 hours of supervised LibriSpeech data. With all three modifications enabled, we achieve a 12% relative reduction in word error rate (WER) on the LibriSpeech test-other set, improving from 10.1% to 8.8%.

Problem

• pseudo-label quality
• automatic speech recognition
• self-supervised learning
• online refinement
• speech representation learning

Innovation

• BEST-RQ
• pseudo-label refinement
• codebook distillation
• PCA-based quantization
• self-supervised speech recognition