Time-Frequency Weighted Losses for Phoneme Reconstruction in DNN-Based Speech Enhancement

2026-06-19

AI Summary

This work addresses the limitation of conventional SDR loss functions in speech enhancement, which treat all time–frequency units uniformly and neglect critical spectral cues essential for phoneme intelligibility. The authors propose a time–frequency weighted SDR loss framework that, for the first time, integrates local acoustic cues—namely speech presence probability, speech-to-interference ratio, and spectral flux—into a differentiable training objective. This approach dynamically emphasizes regions of speech–noise competition while preserving transient structures such as consonantal bursts. By enabling fine-grained weighting tailored to phoneme reconstruction, the method achieves significant improvements in both objective speech enhancement metrics and phoneme recognition accuracy, with notable gains in consonant identification and more effective reconstruction of mid-frequency speech components under moderate signal-to-interference ratios.

Abstract

Conventional training losses for speech enhancement based on the signal-to-distortion ratio (SDR) treat all time-frequency (TF) regions uniformly, overlooking the fine-grained spectral cues that are relevant to specific phoneme intelligibility. We propose a TF weighting framework that modulates the SDR objective based on local speech presence, speech-to-interference ratio (SIR), and spectral flux. By integrating these factors into a differentiable objective, the framework emphasizes TF bins with high speech-noise competition while also accounting for transient cues such as consonant bursts. Experimental results show that our approach improves objective frequency-weighted enhancement metrics, as well as phoneme recognition accuracy, particularly for consonants. Spectral analysis shows better reconstruction of mid-frequency structures at less adverse SIR.

Problem

• speech enhancement
• phoneme reconstruction
• time-frequency weighting
• SDR loss
• speech intelligibility

Innovation

• time-frequency weighting
• phoneme reconstruction
• speech enhancement
• differentiable loss
• spectral flux