Leveraging Mamba with Full-Face Vision for Audio-Visual Speech Enhancement

2025-08-19

AI Summary

Existing Mamba-based speech enhancement models (e.g., SEMamba) are limited to single-speaker scenarios and underperform in multi-talker cocktail-party environments with severe speaker overlap. To address this, we propose the first audio-visual speech enhancement model integrating full-face spatiotemporal visual features with the Mamba architecture. Our method employs a dedicated audio-visual encoder to separately model lip dynamics and facial spatial characteristics, leverages Mamba’s selective state-space modeling for efficient long-range temporal audio dependency capture, and introduces a novel cross-modal fusion mechanism for end-to-end target speech separation and reconstruction. Evaluated under challenging single-channel conditions, the approach significantly improves speech intelligibility and naturalness in multi-speaker settings. On the AVSEC-4 Challenge single-channel track, it achieves state-of-the-art performance—outperforming all monaural baselines across STOI, PESQ, and UTMOS—and ranks first overall.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Speech

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Recent Mamba-based models have shown promise in speech enhancement by efficiently modeling long-range temporal dependencies. However, models like Speech Enhancement Mamba (SEMamba) remain limited to single-speaker scenarios and struggle in complex multi-speaker environments such as the cocktail party problem. To overcome this, we introduce AVSEMamba, an audio-visual speech enhancement model that integrates full-face visual cues with a Mamba-based temporal backbone. By leveraging spatiotemporal visual information, AVSEMamba enables more accurate extraction of target speech in challenging conditions. Evaluated on the AVSEC-4 Challenge development and blind test sets, AVSEMamba outperforms other monaural baselines in speech intelligibility (STOI), perceptual quality (PESQ), and non-intrusive quality (UTMOS), and achieves extbf{1st place} on the monaural leaderboard.

Problem

• Enhancing speech in multi-speaker cocktail party environments
• Integrating full-face visual cues with Mamba-based audio processing
• Improving target speech extraction accuracy in challenging conditions

Innovation

• Integrates full-face visual cues
• Uses Mamba-based temporal backbone
• Leverages spatiotemporal visual information