Online Predictive Coding for Dual-Mode Self-Supervised Speech Model

2026-06-19

AI Summary

This work addresses the instability in training bimodal self-supervised speech models caused by contextual discrepancies between streaming and non-streaming settings, as well as the significant performance gap favoring non-streaming configurations. To mitigate these issues, the authors propose an Online Predictive Coding (OPC) mechanism that regularizes representations through multi-step future prediction, alongside a bimodal layer normalization scheme to enhance training stability. The proposed approach enables joint optimization across both streaming and non-streaming scenarios. Evaluated on LibriSpeech with a 160ms latency constraint, the method reduces the word error rate from 3.65% to 3.40% on test-clean and from 10.15% to 9.65% on test-other, substantially narrowing the performance gap between online and offline inference.

Abstract

Dual-mode self-supervised speech models are pre-trained to handle streaming and non-streaming conditions simultaneously. However, their attention is computed over different context ranges, which often makes optimization difficult. In previous work, we proposed online registers, additional tokens intended to compensate for missing future context in streaming mode, but the gains remained limited. To address these issues, we introduce two improvements for robust dual-mode pre-training: (1) Online Predictive Coding (OPC), which regularizes the registers through multi-step future prediction, and (2) Dual-mode Layer Normalization, which stabilizes optimization. We fine-tune the proposed dual-mode self-supervised speech models for speech recognition on LibriSpeech and WSJ. Results show that OPC consistently reduces the online-offline performance gap; at 160 ms latency on LibriSpeech, word error rates improve from 3.65% to 3.40% on test-clean and from 10.15% to 9.65% on test-other.

Problem

• dual-mode
• self-supervised speech model
• streaming
• optimization
• performance gap

Innovation

• Online Predictive Coding
• Dual-mode Self-Supervised Learning
• Streaming Speech Recognition
• Layer Normalization
• Future Context Prediction