VIB-AVSR: Variational Information Bottleneck for Noise-Robust LLM-Based Audio-Visual Speech Recognition

2026-06-28

AI Summary

This work addresses the significant performance degradation of large language model (LLM)-based audio-visual speech recognition (AVSR) systems in noisy environments, primarily caused by insufficient modeling of noise robustness. To tackle this issue, the study introduces the variational information bottleneck (VIB) into LLM-based AVSR architectures for the first time. Without altering the model structure or augmenting training data, VIB layers are strategically embedded at critical positions within the LLM backbone to regularize multimodal representations. This approach effectively enhances model stability across diverse signal-to-noise ratios and noise types, substantially mitigating performance deterioration and yielding more robust audio-visual speech recognition.

Abstract

Audio-Visual Speech Recognition takes two input modalities, acoustic and visual streams, where visual information from lip movements aids recognition when audio is noisy. Recently, LLM-based AVSR models have emerged as a promising paradigm by connecting pre-trained audio-visual encoders to an LLM, achieving strong results in clean conditions. However, these models are predominantly optimized for clean acoustic conditions, with limited attention to making the LLM backbone robust to noise. No explicit mechanism is employed to produce stable representations under corrupted audio, leading to performance degradation in noisy environments. To address this, we propose VIB-AVSR, which integrates Variational Information Bottleneck layers at targeted positions within the LLM backbone to regularize representations. VIB-AVSR reduces degradation under noisy conditions across multiple SNR levels and noise types, without requiring architectural modifications or additional training data.

Problem

• Audio-Visual Speech Recognition
• Noise Robustness
• Large Language Models
• Representation Stability
• Noisy Environments

Innovation

• Variational Information Bottleneck
• Noise-Robust AVSR
• LLM-based Speech Recognition
• Audio-Visual Fusion
• Representation Regularization