Diffusion-based Unsupervised Audio-visual Speech Enhancement

2024-10-04

arXiv.org

AI Summary

To address the low speech intelligibility of audio-visual speech in noisy environments and the limitations of existing methods—namely their reliance on paired training data or poor real-time performance—this paper proposes an unsupervised audio-visual speech enhancement framework. Our method synergistically integrates a pretrained audio-visual synchronization diffusion model with non-negative matrix factorization (NMF)-based noise modeling, employing a reverse-diffusion-driven iterative posterior sampling scheme to jointly optimize speech estimation and noise parameters—without requiring noisy-clean speech pairs. The key innovations are the first unsupervised diffusion-based audio-visual generative paradigm and end-to-end adaptive noise modeling. Experiments demonstrate that our approach outperforms both conventional audio-only and recent supervised audio-visual speech enhancement (AVSE) methods in terms of signal-to-noise ratio (SNR), perceptual evaluation of speech quality (PESQ), short-time objective intelligibility (STOI), and inference efficiency, achieving a superior trade-off between performance and real-time applicability.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Speech
• Machine Learning: Unsupervised & Self-Supervised Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

This paper proposes a new unsupervised audio-visual speech enhancement (AVSE) approach that combines a diffusion-based audio-visual speech generative model with a non-negative matrix factorization (NMF) noise model. First, the diffusion model is pre-trained on clean speech conditioned on corresponding video data to simulate the speech generative distribution. This pre-trained model is then paired with the NMF-based noise model to estimate clean speech iteratively. Specifically, a diffusion-based posterior sampling approach is implemented within the reverse diffusion process, where after each iteration, a speech estimate is obtained and used to update the noise parameters. Experimental results confirm that the proposed AVSE approach not only outperforms its audio-only counterpart but also generalizes better than a recent supervised-generative AVSE method. Additionally, the new inference algorithm offers a better balance between inference speed and performance compared to the previous diffusion-based method. Code and demo available at: https://jeaneudesayilo.github.io/fast_UdiffSE

Problem

• Speech clarity enhancement
• Noisy environment
• Audio-visual processing

Innovation

• Speech Enhancement
• Diffusion Process
• Non-negative Matrix Factorization (NMF)