Adaptive Convolution for CNN-based Speech Enhancement Models

2025-02-20

AI Summary

To address the limited time-frequency modeling adaptability of CNN-based speech enhancement models, this paper proposes an adaptive convolution module that synergistically integrates frame-level causal dynamic convolution with lightweight attention, generating time-varying convolutional kernels per frame to enable history-aware dynamic kernel weight aggregation. Based on this, we design the ultra-lightweight AdaptCRN model (only 0.14M parameters), featuring a CNN-RNN hybrid encoder-decoder architecture. Evaluated on multiple CNN baselines, AdaptCRN achieves significant improvements in PESQ (+0.32) and STOI (+1.8%), outperforming state-of-the-art models with over 3.5× more parameters, while incurring negligible computational overhead. To the best of our knowledge, this work introduces the first frame-level dynamic convolution architecture for speech enhancement, uniquely balancing strong input-dependent adaptivity with exceptionally low computational complexity.

Technology Category

• Cognitive Modeling & Cognitive Systems: Adaptive Behavior
• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: Speech

Application Category

• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Deep learning-based speech enhancement methods have significantly improved speech quality and intelligibility. Convolutional neural networks (CNNs) have been proven to be essential components of many high-performance models. In this paper, we introduce adaptive convolution, an efficient and versatile convolutional module that enhances the model's capability to adaptively represent speech signals. Adaptive convolution performs frame-wise causal dynamic convolution, generating time-varying kernels for each frame by assembling multiple parallel candidate kernels. A Lightweight attention mechanism leverages both current and historical information to assign adaptive weights to each candidate kernel, guiding their aggregation. This enables the convolution operation to adapt to frame-level speech spectral features, leading to more efficient extraction and reconstruction. Experimental results on various CNN-based models demonstrate that adaptive convolution significantly improves the performance with negligible increases in computational complexity, especially for lightweight models. Furthermore, we propose the adaptive convolutional recurrent network (AdaptCRN), an ultra-lightweight model that incorporates adaptive convolution and an efficient encoder-decoder design, achieving superior performance compared to models with similar or even higher computational costs.

Problem

• Enhance CNN-based speech enhancement models
• Introduce adaptive convolution for speech signals
• Improve model performance with minimal complexity

Innovation

• Adaptive convolution for speech enhancement
• Time-varying kernels with attention mechanism
• Adaptive convolutional recurrent network design