Scalable Frameworks for Real-World Audio-Visual Speech Recognition

2025-12-15

AI Summary

In real-world scenarios, audio-visual speech recognition (AVSR) suffers severe performance degradation under acoustic noise and visual occlusion, hindering practical deployment. To address this, we propose a robust and scalable AVSR framework tailored for open environments, jointly optimizing representation, architecture, and system design. First, we introduce hierarchical noise- and occlusion-robust audio-visual unified representation learning—a novel paradigm for joint multimodal representation. Second, we design an input-adaptive multimodal computation allocation mechanism to dynamically distribute computational resources across modalities. Third, we establish a modular, plug-and-play collaborative system architecture supporting seamless integration of large language models (LLMs) and audio-visual foundation models. Our method incorporates cross-modal feature alignment and fusion, dynamic resource scheduling, and extensible functional interfaces. Evaluated under diverse noise and occlusion conditions, our approach achieves a 32% relative reduction in word error rate (WER) and a 2.1× speedup in inference latency, significantly enhancing robustness and scalability in realistic settings.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The practical deployment of Audio-Visual Speech Recognition (AVSR) systems is fundamentally challenged by significant performance degradation in real-world environments, characterized by unpredictable acoustic noise and visual interference. This dissertation posits that a systematic, hierarchical approach is essential to overcome these challenges, achieving the robust scalability at the representation, architecture, and system levels. At the representation level, we investigate methods for building a unified model that learns audio-visual features inherently robust to diverse real-world corruptions, thereby enabling generalization to new environments without specialized modules. To address architectural scalability, we explore how to efficiently expand model capacity while ensuring the adaptive and reliable use of multimodal inputs, developing a framework that intelligently allocates computational resources based on the input characteristics. Finally, at the system level, we present methods to expand the system's functionality through modular integration with large-scale foundation models, leveraging their powerful cognitive and generative capabilities to maximize final recognition accuracy. By systematically providing solutions at each of these three levels, this dissertation aims to build a next-generation, robust, and scalable AVSR system with high reliability in real-world applications.

Problem

• Addresses performance degradation in real-world AVSR due to noise and interference.
• Develops hierarchical methods for robust representation, architecture, and system scalability.
• Enhances reliability through unified models, adaptive computation, and foundation model integration.

Innovation

• Unified robust audio-visual feature learning
• Adaptive computational resource allocation framework
• Modular integration with large foundation models