Combining Deterministic Enhanced Conditions with Dual-Streaming Encoding for Diffusion-Based Speech Enhancement

2025-05-20

AI Summary

To address information distortion in diffusion-based speech enhancement caused by unreliable noise conditioning, this paper proposes DERDM-SE—a deterministic-enhancement-guided repair diffusion model for speech enhancement. First, a deterministic enhancement frontend—comprising a UNet backbone and a fine-grained time-frequency calibration module—is introduced to generate high-fidelity prior features. Second, a Repair-Diffusion dual-stream encoder is designed to separately process the deterministic prior and noisy observation, with gated conditional fusion enabling coarse-to-fine collaborative modeling. Crucially, DERDM-SE pioneers a deterministic–noisy dual-conditioning mechanism for diffusion modeling. Evaluated on CHiME4, it achieves significant improvements in PESQ (+0.32) and STOI (+1.8%), demonstrating both superior objective metrics and enhanced subjective perceptual stability—outperforming existing diffusion-based speech enhancement methods.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Diffusion-based speech enhancement (SE) models need to incorporate correct prior knowledge as reliable conditions to generate accurate predictions. However, providing reliable conditions using noisy features is challenging. One solution is to use features enhanced by deterministic methods as conditions. However, the information distortion and loss caused by deterministic methods might affect the diffusion process. In this paper, we first investigate the effects of using different deterministic SE models as conditions for diffusion. We validate two conditions depending on whether the noisy feature was used as part of the condition: one using only the deterministic feature (deterministic-only), and the other using both deterministic and noisy features (deterministic-noisy). Preliminary investigation found that using deterministic enhanced conditions improves hearing experiences on real data, while the choice between using deterministic-only or deterministic-noisy conditions depends on the deterministic models. Based on these findings, we propose a dual-streaming encoding Repair-Diffusion Model for SE (DERDM-SE) to more effectively utilize both conditions. Moreover, we found that fine-grained deterministic models have greater potential in objective evaluation metrics, while UNet-based deterministic models provide more stable diffusion performance. Therefore, in the DERDM-SE, we propose a deterministic model that combines coarse- and fine-grained processing. Experimental results on CHiME4 show that the proposed models effectively leverage deterministic models to achieve better SE evaluation scores, along with more stable performance compared to other diffusion-based SE models.

Problem

• Investigates effects of deterministic SE models as diffusion conditions
• Proposes dual-stream encoding to utilize deterministic and noisy features
• Combines coarse- and fine-grained processing for stable diffusion performance

Innovation

• Combines deterministic and noisy features
• Uses dual-streaming encoding Repair-Diffusion Model
• Integrates coarse- and fine-grained processing