Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Models

2025-11-10

AI Summary

Existing LLM-based speech recognition approaches treat audio-only (ASR), visual-only (VSR), and audio-visual (AVSR) speech recognition as separate tasks, necessitating multiple dedicated models—leading to high computational overhead and an inability to capture cross-task synergies. Moreover, fixed-rate token compression restricts flexibility in trading off accuracy for efficiency. This paper proposes Omni-AVSR, the first unified audio-visual speech recognition framework supporting joint modeling of all three tasks and elastic inference. We innovatively introduce Matryoshka representation learning into multimodal speech recognition and design three LoRA-based adaptive strategies to enable multi-granularity representation learning and parameter-efficient fine-tuning. Evaluated on LRS2 and LRS3, Omni-AVSR achieves state-of-the-art performance across ASR, VSR, and AVSR, significantly reduces training and deployment costs, and demonstrates strong robustness under noisy conditions.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large language models (LLMs) have recently achieved impressive results in speech recognition across multiple modalities, including Auditory Speech Recognition (ASR), Visual Speech Recognition (VSR), and Audio-Visual Speech Recognition (AVSR). Despite this progress, current LLM-based approaches typically address each task independently, training separate models that raise computational and deployment resource use while missing potential cross-task synergies. They also rely on fixed-rate token compression, which restricts flexibility in balancing accuracy with efficiency. These limitations highlight the need for a unified framework that can support ASR, VSR, and AVSR while enabling elastic inference. To this end, we present Omni-AVSR, a unified audio-visual LLM that combines efficient multi-granularity training with parameter-efficient adaptation. Specifically, we adapt the matryoshka representation learning paradigm to efficiently train across multiple audio and visual granularities, reducing its inherent training resource use. Furthermore, we explore three LoRA-based strategies for adapting the backbone LLM, balancing shared and task-specific specialization. Experiments on LRS2 and LRS3 show that Omni-AVSR achieves comparable or superior accuracy to state-of-the-art baselines while training a single model at substantially lower training and deployment resource use. The model also remains robust under acoustic noise, and we analyze its scaling behavior as LLM size increases, providing insights into the trade-off between performance and efficiency.

Problem

• Unifying multimodal speech recognition tasks under single framework
• Enabling flexible accuracy-efficiency tradeoffs through elastic inference
• Reducing computational costs while maintaining cross-task performance robustness

Innovation

• Unified multimodal LLM for ASR, VSR, AVSR
• Multi-granularity training with matryoshka representation learning
• LoRA-based adaptation strategies for parameter efficiency