Audio-visual Contrastive Alignment for Diffusion-based Visual-conditioned Speech Enhancement

2026-06-16

AI Summary

This work addresses the limited utilization of visual cues—such as lip movements—in existing audio-visual speech enhancement methods under noisy conditions, which stems from insufficient cross-modal alignment. To overcome this limitation, the study introduces contrastive learning into a diffusion-based speech enhancement framework for the first time. While preserving the posterior sampling structure, the authors propose a visual-conditioned diffusion model integrated with a cross-attention mechanism and design a contrastive audio-visual alignment loss to explicitly encourage semantic alignment between acoustic and visual features. The proposed approach significantly improves interference suppression, signal reconstruction accuracy, and perceptual quality under both matched and mismatched test conditions, with the most pronounced gains observed in low signal-to-noise ratio scenarios.

Abstract

Audio-visual speech enhancement (AVSE) exploits visual cues such as lip movements to recover speech in noisy environments. Recent work introduced diffusion-based unsupervised AVSE, where a speech diffusion model conditioned on visual features via cross-attention is trained and used as a data-driven prior for posterior sampling-based speech enhancement. Despite promising performance over its audio-only counterpart, the impact of explicitly enforcing cross-modal alignment in the fusion remains unclear. In this work, we propose to augment the diffusion training objective with a contrastive audio-visual loss to encourage stronger use of visual information while keeping the posterior sampling framework unchanged. Experiments across matched and mismatched test data show consistent improvements in interference suppression, signal reconstruction, and perceptual quality, with the largest gains at low SNRs. Code is available at https://github.com/ cexauce/AV-CA-DiffUSE

Problem

• audio-visual speech enhancement
• diffusion model
• cross-modal alignment
• contrastive learning
• visual-conditioned speech enhancement

Innovation

• audio-visual contrastive alignment
• diffusion-based speech enhancement
• visual-conditioned speech enhancement
• cross-modal alignment
• unsupervised AVSE