Diffusion Large Language Models for Visual Speech Recognition

2026-05-27

AI Summary

This work addresses the limitations of traditional visual speech recognition (VSR) systems, which rely on autoregressive decoding and often produce premature predictions for visually ambiguous tokens under insufficient contextual cues. To overcome this, the study introduces DLLM-VSR, the first framework to integrate a diffusion-based large language model into VSR. It enables non-autoregressive, flexible-order decoding through iterative masked denoising, complemented by a confidence-driven progressive unmasking strategy and bidirectional contextual modeling. To handle target sequence length uncertainty, the authors propose a two-stage masked denoising training scheme along with a length-guided candidate decoding mechanism. Evaluated solely on the LRS3 dataset with standard annotations, the method achieves a word error rate (WER) of 19.5%, establishing a new state-of-the-art performance.

Abstract

Existing Visual Speech Recognition (VSR) systems commonly rely on left-to-right autoregressive decoding, which can force premature decisions on visually ambiguous tokens before sufficient context is available. We propose DLLM-VSR, to the best of our knowledge, the first Diffusion Large Language Model (DLLM)-based VSR framework, formulating transcription as iterative masked denoising with flexible-order decoding. With confidence-based unmasking, DLLM-VSR commits high-confidence positions early and uses the committed tokens as bidirectional context to refine ambiguous ones. To adapt DLLMs to VSR, we introduce a two-stage masked-denoising training strategy that separates visual-to-text content alignment from length modeling. We further observe a performance gap with oracle-length decoding, which assumes access to the true transcript length, indicating that reducing target-length uncertainty can improve DLLM-based VSR. To reduce this gap, we develop length-guided candidate decoding, which uses video duration to construct plausible transcript-length hypotheses, decodes under multiple hypotheses, and reranks candidates using length plausibility and decoding confidence. The proposed method achieves a state-of-the-art WER of 19.5\% on LRS3 using only its labeled training data.

Problem

• Visual Speech Recognition
• autoregressive decoding
• visually ambiguous tokens
• context availability
• transcript length uncertainty

Innovation

• Diffusion Large Language Model
• Visual Speech Recognition
• Masked Denoising
• Length-Guided Decoding
• Confidence-Based Unmasking