RT-SEMamba: Real-Time Speech Enhancement Mamba via Progressive Knowledge Distillation

2026-08-12

AI Summary

This work addresses the longstanding challenge in real-time speech enhancement of simultaneously achieving high audio quality and low latency. To this end, it introduces Mamba—a state space model—into this domain for the first time, proposing a fully causal architecture built upon causal time-frequency Mamba blocks. To mitigate computational overhead, the authors devise a progressive knowledge distillation strategy that jointly transfers both spectral outputs and intermediate representations from an 8-layer teacher model to a single-layer student model. Evaluated on the Voicebank-DEMAND dataset, the distilled student model achieves a PESQ score of 3.18—improved from 3.06—with only 25 ms of latency and a 2.75× speedup in inference, significantly outperforming current state-of-the-art approaches.

Abstract

We present RT-SEMamba, a fully causal speech enhancement (SE) model built upon causal time-frequency Mamba blocks. Unlike Transformer-based architectures that rely on a growing key-value cache, Mamba propagates a fixed-size recurrent state per layer, enabling memory- and bandwidth-efficient long-form inference. We further introduce a progressive knowledge distillation (KD) strategy that compresses an 8-layer teacher into a shallow 1-layer student by jointly distilling complex spectral outputs and intermediate representations. On Voicebank-DEMAND, the 8-layer RT-SEMamba achieves 3.32 PESQ with a 25 ms algorithmic latency constraint, and the distilled 1-layer student improves over a naive 1-layer baseline from 3.06 to 3.18 PESQ while preserving the same steady-state RTF, delivering a 2.75x speedup over the teacher. These results demonstrate that state-space models with progressive KD provide a competitive quality-latency trade-off for real-time SE.

Problem

• real-time speech enhancement
• low latency
• efficient inference
• quality-latency trade-off

Innovation

• Mamba
• speech enhancement
• knowledge distillation
• real-time inference
• state-space models