Leveraging Large Language Models in Visual Speech Recognition: Model Scaling, Context-Aware Decoding, and Iterative Polishing

2025-05-27

AI Summary

This paper addresses the underexploited potential of large language models (LLMs) in visual speech recognition (VSR). We propose a three-stage LLM-cooperative paradigm: (1) empirically uncovering the LLM scaling law for VSR tasks; (2) designing a context-guided decoding mechanism that dynamically injects both visual and linguistic contextual cues; and (3) establishing a multi-round semantic iterative refinement framework to enable cross-modal alignment fine-tuning and self-correcting decoding. Our approach overcomes the decoding bottlenecks inherent in conventional end-to-end VSR systems. Evaluated on standard benchmarks—including LRW and LRS3—our method achieves a 12.6% relative reduction in word error rate over the state of the art. These results robustly demonstrate the efficacy and generalizability of LLMs in leveraging strong linguistic priors for purely vision-based lip reading, even without audio input.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Visual Speech Recognition (VSR) transcribes speech by analyzing lip movements. Recently, Large Language Models (LLMs) have been integrated into VSR systems, leading to notable performance improvements. However, the potential of LLMs has not been extensively studied, and how to effectively utilize LLMs in VSR tasks remains unexplored. This paper systematically explores how to better leverage LLMs for VSR tasks and provides three key contributions: (1) Scaling Test: We study how the LLM size affects VSR performance, confirming a scaling law in the VSR task. (2) Context-Aware Decoding: We add contextual text to guide the LLM decoding, improving recognition accuracy. (3) Iterative Polishing: We propose iteratively refining LLM outputs, progressively reducing recognition errors. Extensive experiments demonstrate that by these designs, the great potential of LLMs can be largely harnessed, leading to significant VSR performance improvement.

Problem

• Exploring LLM size impact on Visual Speech Recognition performance
• Improving VSR accuracy with context-aware LLM decoding
• Reducing recognition errors via iterative LLM output polishing

Innovation

• Scaling LLM size for VSR performance
• Context-aware decoding with text guidance
• Iterative polishing to reduce errors