AUREXA-SE: Audio-Visual Unified Representation Exchange Architecture with Cross-Attention and Squeezeformer for Speech Enhancement

2025-10-06

AI Summary

This work addresses audio-visual speech enhancement by proposing a progressive dual-modal framework that jointly models raw audio waveforms and lip-motion visual cues. Methodologically, it introduces a bidirectional cross-attention mechanism for deep cross-modal interaction and employs lightweight stacked Squeezeformer modules to capture temporal dependencies. The architecture integrates a 1D convolutional encoder, Swin Transformer V2, and a multi-stage feature exchange module to unify multimodal representation learning and waveform reconstruction. Key contributions include the construction of a shared latent representation space across modalities and an efficient joint modeling of temporal dynamics and cross-modal correlations. Evaluated on standard benchmarks, the method achieves state-of-the-art performance: STOI = 0.516, PESQ = 1.323, and SI-SDR = −4.322 dB—substantially outperforming existing approaches.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Speech

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

In this paper, we propose AUREXA-SE (Audio-Visual Unified Representation Exchange Architecture with Cross-Attention and Squeezeformer for Speech Enhancement), a progressive bimodal framework tailored for audio-visual speech enhancement (AVSE). AUREXA-SE jointly leverages raw audio waveforms and visual cues by employing a U-Net-based 1D convolutional encoder for audio and a Swin Transformer V2 for efficient and expressive visual feature extraction. Central to the architecture is a novel bidirectional cross-attention mechanism, which facilitates deep contextual fusion between modalities, enabling rich and complementary representation learning. To capture temporal dependencies within the fused embeddings, a stack of lightweight Squeezeformer blocks combining convolutional and attention modules is introduced. The enhanced embeddings are then decoded via a U-Net-style decoder for direct waveform reconstruction, ensuring perceptually consistent and intelligible speech output. Experimental evaluations demonstrate the effectiveness of AUREXA-SE, achieving significant performance improvements over noisy baselines, with STOI of 0.516, PESQ of 1.323, and SI-SDR of -4.322 dB. The source code of AUREXA-SE is available at https://github.com/mtanveer1/AVSEC-4-Challenge-2025.

Problem

• Enhancing noisy speech using audio-visual fusion and cross-attention mechanisms
• Extracting complementary features from raw audio waveforms and visual cues
• Improving speech intelligibility and quality through multimodal representation learning

Innovation

• U-Net encoder processes raw audio waveforms
• Swin Transformer extracts expressive visual features
• Squeezeformer blocks capture temporal dependencies in embeddings