One Model, Many Latencies: Universal Speech Enhancement for Diverse Real-Time Applications

2026-06-24

AI Summary

This work addresses the high deployment cost of existing real-time speech enhancement methods, which typically require separate models trained for each specific latency constraint. To overcome this limitation, the authors propose a unified real-time speech enhancement framework featuring a configurable look-ahead frame mechanism that flexibly controls algorithmic latency. The architecture incorporates parallel convolutional structures tailored to diverse look-ahead settings and integrates an early-exit mechanism to dynamically adjust computational overhead. A two-stage training strategy is introduced: first training a shared decoder, then transferring knowledge to a multi-decoder configuration, thereby effectively narrowing the performance gap between the universal model and specialized counterparts. This approach enables a single model to meet varied latency budgets without retraining, significantly enhancing deployment flexibility while maintaining high performance.

Abstract

Different real-time speech applications impose distinct latency budgets, often requiring separately trained enhancement models for each scenario. In this paper, we propose a one-for-all, real-time universal speech enhancement model that provides explicit control over both algorithmic and computational latency. Algorithmic latency is flexibly adjusted via configurable look-ahead frames. To avoid learning inefficiency caused by varying padding configurations, we introduce parallel convolutional layers corresponding to different look-ahead settings. Computational latency is controlled through an early-exit mechanism, enabling inference at different network depths. To narrow the performance gap between specialized and flexible models, we propose a two-stage training strategy with a shared-to-multiple decoder transition. Overall, the proposed framework enables a single model to be deployed across diverse latency budgets without retraining separate models.

Problem

• speech enhancement
• latency constraints
• real-time applications
• model specialization
• universal model

Innovation

• universal speech enhancement
• latency control
• parallel convolutional layers
• early-exit mechanism
• two-stage training