Online Register for Dual-Mode Self-Supervised Speech Models: Mitigating The Lack of Future Context

2026-02-27

AI Summary

This work addresses the significant performance degradation of bimodal self-supervised models in streaming speech processing due to the absence of future contextual information. To mitigate this limitation without introducing additional latency, the authors propose an “online register” mechanism that appends learnable virtual placeholders after each audio chunk. These registers are trained with a future prediction loss to capture forward-looking semantic cues, effectively compensating for the missing context in real-time processing. The approach substantially narrows the performance gap between offline and online inference regimes. Empirical evaluations on LibriSpeech and out-of-domain benchmarks demonstrate its efficacy, achieving a 3.4% relative word error rate reduction under a 160-millisecond chunking configuration.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising

Abstract

Dual-mode self-supervised speech models (S3Ms), which jointly pre-trained in the offline and online mode, suffer from attention mismatch in streaming scenarios due to missing future context. To address this challenge, we proposed online registers, learnable tokens appended to each chunk in online mode. These tokens act as virtual placeholders for unseen future frames, enabling the model to compensate for missing context without introducing additional latency. Furthermore, we introduce a future prediction loss that explicitly guides the registers to capture predictive cues, thereby enriching their ability to retain future information. Experiments on LibriSpeech, and out-of-domain benchmarks demonstrate that online registers consistently reduce the performance gap between offline and online modes, achieving a 3.4% relative improvement on LibriSpeech with 160 ms chunks, especially in low-latency settings.

Problem

• dual-mode self-supervised speech models
• streaming scenarios
• future context
• attention mismatch
• online inference

Innovation

• online registers
• dual-mode self-supervised speech models
• future context prediction
• streaming speech recognition
• low-latency modeling