Beyond U-Net: A Latent-Representation-Aligned Skip-Free Backbone for Flow-Matching Speech Enhancement

2026-06-23

AI Summary

This work addresses a key limitation in existing U-Net-based flow-matching approaches for speech enhancement, where skip connections often propagate noisy low-level features to the decoder, compromising both enhancement quality and real-time performance. To overcome this, the authors propose a novel encoder-decoder architecture that eliminates skip connections entirely and instead aligns the flow-matching process with clean speech latent representations extracted from a frozen Descript audio codec. This alignment mechanism preserves compact semantic information while preventing noise leakage from encoder to decoder. Remarkably, the method achieves high-fidelity reconstruction with only five function evaluations. Experimental results on WSJ0-CHiME3 and VoiceBank-DEMAND demonstrate substantial improvements in PESQ and perceptual quality over state-of-the-art methods, with particularly pronounced gains on VoiceBank-DEMAND.

Abstract

Generative models, particularly diffusion and score-based approaches, have recently achieved strong performance in speech enhancement, but their iterative sampling process limits real-time deployment. Flow Matching offers an efficient alternative by transporting noisy speech toward clean speech through an ordinary differential equation with few function evaluations. In this work, we propose a skip-free encoder-decoder backbone for flow-matching speech enhancement, guided by Latent Representation Alignment (LRA). Instead of relying on U-Net skip connections, which may transfer noise-correlated low-level features to the decoder, the proposed model aligns its bottleneck and decoder representations with clean latent features extracted from a frozen Descript Audio Codec encoder-decoder without quantization. This codec-aligned supervision promotes compact clean-speech representations while preserving efficient few-step inference. Experiments on WSJ0-CHiME3 and VoiceBank-DEMAND show improved PESQ and perceptual quality, especially on VoiceBank-DEMAND, using only five function evaluations.

Problem

• speech enhancement
• flow matching
• U-Net
• latent representation
• real-time deployment

Innovation

• Flow Matching
• Skip-Free Backbone
• Latent Representation Alignment
• Speech Enhancement
• Generative Modeling