Conditional Latent Diffusion-Based Speech Enhancement Via Dual Context Learning

2025-01-17

AI Summary

Existing diffusion models for speech enhancement suffer from high computational overhead, insufficient modeling of fine-grained acoustic details, and poor generalization to unseen noise types. To address these limitations, we propose a synergistic framework comprising a conditional Latent Diffusion Model (cLDM) and Dual Context Learning (DCL). First, a VAE compresses mel-spectrograms into a low-dimensional latent space. Then, for the first time in speech enhancement, we formulate a bidirectional diffusion process in this latent space to jointly model the clean speech–noise joint distribution. The DCL mechanism integrates local temporal and global semantic contexts, significantly improving robustness to out-of-domain noise (e.g., DNS-Challenge unseen noise). Experiments demonstrate that our method surpasses state-of-the-art diffusion-based approaches on PESQ and STOI metrics, while reducing inference steps by 40%, thereby achieving both computational efficiency and strong generalization capability.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Speech

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recently, the application of diffusion probabilistic models has advanced speech enhancement through generative approaches. However, existing diffusion-based methods have focused on the generation process in high-dimensional waveform or spectral domains, leading to increased generation complexity and slower inference speeds. Additionally, these methods have primarily modelled clean speech distributions, with limited exploration of noise distributions, thereby constraining the discriminative capability of diffusion models for speech enhancement. To address these issues, we propose a novel approach that integrates a conditional latent diffusion model (cLDM) with dual-context learning (DCL). Our method utilizes a variational autoencoder (VAE) to compress mel-spectrograms into a low-dimensional latent space. We then apply cLDM to transform the latent representations of both clean speech and background noise into Gaussian noise by the DCL process, and a parameterized model is trained to reverse this process, conditioned on noisy latent representations and text embeddings. By operating in a lower-dimensional space, the latent representations reduce the complexity of the generation process, while the DCL process enhances the model's ability to handle diverse and unseen noise environments. Our experiments demonstrate the strong performance of the proposed approach compared to existing diffusion-based methods, even with fewer iterative steps, and highlight the superior generalization capability of our models to out-of-domain noise datasets (https://github.com/modelscope/ClearerVoice-Studio).

Problem

• Diffusion Models
• Speech Quality Improvement
• Computational Efficiency

Innovation

• cLDM
• DCL
• VAE