Speech Enhancement Based on Drifting Models

2026-04-27

AI Summary

This work addresses the inefficiency of multi-step iterative denoising and the difficulty in leveraging unpaired data in speech enhancement by proposing DriftSE, a novel framework that introduces drift field modeling to this domain for the first time. Formulating denoising as an equilibrium problem, DriftSE employs a drift field–driven mapping function to directly approximate the clean speech distribution via forward pushforward, enabling both single-step inference and training with unpaired data. The framework encompasses two variants: direct mapping and Gaussian prior–conditioned generation. Evaluated on the VoiceBank-DEMAND benchmark, DriftSE surpasses existing diffusion-based models, achieving highly efficient and high-fidelity single-step speech enhancement and establishing a new paradigm for generative modeling in speech processing.

Technology Category

• Natural Language Processing: Speech
• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Privacy-enhancing technologies

Abstract

We propose Speech Enhancement based on Drifting Models (DriftSE), a novel generative framework that formulates denoising as an equilibrium problem. Rather than relying on iterative sampling, DriftSE natively achieves one-step inference by evolving the pushforward distribution of a mapping function to directly match the clean speech distribution. This evolution is driven by a Drifting Field, a learned correction vector that guides samples toward the high-density regions of the clean distribution, which naturally facilitates training on unpaired data by matching distributions rather than paired samples. We investigate the framework under two formulations: a direct mapping from the noisy observation, and a stochastic conditional generative model from a Gaussian prior. Experiments on the VoiceBank-DEMAND benchmark demonstrate that DriftSE achieves high-fidelity enhancement in a single step, outperforming multi-step diffusion baselines and establishing a new paradigm for speech enhancement.

Problem

• Speech Enhancement
• One-step Inference
• Unpaired Data
• Distribution Matching
• Generative Modeling

Innovation

• Drifting Models
• One-step Inference
• Distribution Matching
• Speech Enhancement
• Generative Framework