Adapter-Based Multi-Agent AVSR Extension for Pre-Trained ASR Models

2025-02-03

AI Summary

Audio-visual speech recognition (AVSR) under diverse and varying noise conditions remains challenging, particularly when visual input is unreliable or unavailable. Method: This paper proposes a lightweight, adaptive visual enhancement framework built upon the pre-trained Whisper model. It integrates LoRA adapters with a dedicated visual branch, forming a novel multi-agent adapter architecture: distinct adapter groups are trained for specific noise types and intensities, dynamically scheduled by a noise-scene classifier; crucially, the system seamlessly degrades to audio-only ASR when visual input is absent. Contribution/Results: The method fine-tunes only 0.7% of Whisper’s parameters—reducing trainable parameters by 88.5% versus full fine-tuning—yet achieves near-state-of-the-art accuracy across most noisy scenarios. It delivers strong robustness to heterogeneous acoustic distortions, minimal computational overhead, and inherent scalability to new noise conditions and modalities.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

We present an approach to Audio-Visual Speech Recognition that builds on a pre-trained Whisper model. To infuse visual information into this audio-only model, we extend it with an AV fusion module and LoRa adapters, one of the most up-to-date adapter approaches. One advantage of adapter-based approaches, is that only a relatively small number of parameters are trained, while the basic model remains unchanged. Common AVSR approaches train single models to handle several noise categories and noise levels simultaneously. Taking advantage of the lightweight nature of adapter approaches, we train noise-scenario-specific adapter-sets, each covering individual noise-categories or a specific noise-level range. The most suitable adapter-set is selected by previously classifying the noise-scenario. This enables our models to achieve an optimum coverage across different noise-categories and noise-levels, while training only a minimum number of parameters. Compared to a full fine-tuning approach with SOTA performance our models achieve almost comparable results over the majority of the tested noise-categories and noise-levels, with up to 88.5% less trainable parameters. Our approach can be extended by further noise-specific adapter-sets to cover additional noise scenarios. It is also possible to utilize the underlying powerful ASR model when no visual information is available, as it remains unchanged.

Problem

• Enhancing ASR with visual data
• Reducing parameters via adapters
• Optimizing noise-specific speech recognition

Innovation

• Adapter-based AVSR extension
• LoRa adapters integration
• Noise-scenario-specific adapter-sets