Bridging Self-Supervised Learning and Speech Enhancement: A Wav2Vec2-Conditioned Framework

2026-06-21

AI Summary

This work addresses the lack of linguistic-level guidance in existing diffusion models for speech enhancement, which often results in reconstructed speech with insufficient semantic coherence. To remedy this, the study proposes a novel approach that leverages frozen self-supervised features from wav2vec 2.0 as conditioning signals, injected into the bottleneck layer of a diffusion U-Net via Feature-wise Linear Modulation (FiLM). An exponential smoothing mechanism is introduced to aggregate FiLM parameters over time, enabling efficient temporal compression and more effective denoising guidance. Evaluated on the VoiceBank-DEMAND and LibriMix datasets, the method significantly outperforms unconditional baselines, achieving a consistent improvement across multiple objective metrics—including a 0.4 absolute gain in PESQ, along with notable enhancements in STOI, SI-SDR, and DNSMOS scores.

Abstract

Diffusion models show potential for speech enhancement but lack linguistic guidance. We condition a diffusion-based model on wav2vec 2.0 features from noisy input, injected at the U-Net bottleneck via Feature-wise Linear Modulation (FiLM). Phonetic representations from wav2vec 2.0 features of degraded speech, anchor the reverse diffusion process. While a frozen wav2vec 2.0 encoder extracts features, a learned FiLM generator produces scale and shift parameters modulating the bottleneck with minimal overhead. Motivated by the optimal Bayesian causal estimator under a linear-Gaussian state-space model, FiLM coefficients are aggregated via exponential smoothing for temporal compression. Evaluation on VoiceBank-DEMAND and LibriMix shows competitive performance against the unconditioned baseline in PESQ, STOI, SI-SDR and DNSMOS. We consistently record an improvement of 0.4 on PESQ score, suggesting self-supervised representations effectively condition diffusion-based speech enhancement.

Problem

• speech enhancement
• diffusion models
• self-supervised learning
• linguistic guidance
• wav2vec 2.0

Innovation

• diffusion models
• self-supervised learning
• wav2vec 2.0
• Feature-wise Linear Modulation
• speech enhancement