TF-MoE: Time-Frequency Mixture-of-Experts for Efficient Speech Separation

2026-06-28

AI Summary

This work addresses the high computational cost of speech separation models when deployed on edge devices by proposing a sparse mixture-of-experts (MoE) framework that alternately activates experts along time and frequency dimensions, substantially increasing model capacity with negligible additional inference overhead. Built upon a Mel-band-partitioned Conformer backbone, the approach integrates time- and frequency-alternating MoE modules to achieve efficient yet accurate separation. Evaluated on the Libri2Mix dataset, the method attains a signal-to-distortion ratio (SDR) 3.8 dB higher than BSRNN at only 4.1 GMACs/s, significantly outperforming existing low-complexity approaches.

Abstract

Recent advances in speech separation (SS) have led to compact front-end models with small parameter sizes, yet their high computational cost remains a major barrier for deployment on edge devices. To address this, we propose TF-MoE, a sparse Mixture-of-Experts (MoE) framework that enhances model capacity with almost no increase in inference cost. Our method introduces dynamic expert specialization in time and frequency dimensions through alternating time-wise and frequency-wise MoE modules, each dynamically selecting experts per frame or mel band. Built upon a mel-band-splitting Conformer backbone, TF-MoE achieves strong performance on SS tasks under low-compute settings. Experimental results demonstrate that TF-MoE consistently improves separation performance under computation cost constraints, outperforming BSRNN by +3.8 dB SDR on Libri2Mix with comparable 4.1 GMACs/s inference cost. This positions TF-MoE as a promising candidate for edge-device deployment.

Problem

• speech separation
• computational cost
• edge devices
• model efficiency

Innovation

• Mixture-of-Experts
• time-frequency modeling
• speech separation
• edge deployment
• sparse activation