Pay Attention to CTC: Fast and Robust Pseudo-Labelling for Unified Speech Recognition

2026-02-22

AI Summary

This work addresses key challenges in semi-supervised unified speech recognition (USR), where high-cost autoregressive pseudo-labeling and decoupled supervision between CTC and attention branches often lead to error accumulation under distribution shifts. To overcome these issues, the authors propose a CTC-driven teacher forcing mechanism that, for the first time, directly leverages greedy-decoded CTC pseudo-labels as targets for the attention branch, enabling synchronized training of both branches without expensive beam search. Additionally, a hybrid sampling strategy is introduced to mitigate exposure bias in the decoder. The proposed approach achieves state-of-the-art performance on LRS3, LRS2, and WildVSR benchmarks, halves training time, and significantly enhances robustness to out-of-distribution data, including long sequences, noisy inputs, and unseen domains.

Technology Category

• Machine Learning: Semi-Supervised Learning
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Unified Speech Recognition (USR) has emerged as a semi-supervised framework for training a single model for audio, visual, and audiovisual speech recognition, achieving state-of-the-art results on in-distribution benchmarks. However, its reliance on autoregressive pseudo-labelling makes training expensive, while its decoupled supervision of CTC and attention branches increases susceptibility to self-reinforcing errors, particularly under distribution shifts involving longer sequences, noise, or unseen domains. We propose CTC-driven teacher forcing, where greedily decoded CTC pseudo-labels are fed into the decoder to generate attention targets in a single forward pass. Although these can be globally incoherent, in the pseudo-labelling setting they enable efficient and effective knowledge transfer. Because CTC and CTC-driven attention pseudo-labels have the same length, the decoder can predict both simultaneously, benefiting from the robustness of CTC and the expressiveness of attention without costly beam search. We further propose mixed sampling to mitigate the exposure bias of the decoder relying solely on CTC inputs. The resulting method, USR 2.0, halves training time, improves robustness to out-of-distribution inputs, and achieves state-of-the-art results on LRS3, LRS2, and WildVSR, surpassing USR and modality-specific self-supervised baselines.

Problem

• Unified Speech Recognition
• pseudo-labelling
• CTC
• distribution shift
• robustness

Innovation

• CTC-driven teacher forcing
• pseudo-labelling
• unified speech recognition
• mixed sampling
• exposure bias mitigation