Breaking the Quality--Intelligibility Trade-off in Streaming Target Speaker Extraction via Deep-Feature-Anchored Preference Optimization

2026-07-11

AI Summary

This work addresses the trade-off between speech perceptual quality and intelligibility in streaming target speaker extraction, where enhancing quality often degrades intelligibility. To mitigate this, the authors propose a Direct Preference Optimization (DPO) method leveraging deep acoustic features from WavLM, introducing for the first time cosine similarity in deep feature space as an anchor for constructing preference pairs. Combined with an enlarged Conformer convolution kernel, this approach enables joint modeling of speech content and speaker identity, effectively preventing reward hacking. Evaluated under a 560 ms streaming chunk setting, the method reduces word error rate from 0.138 to 0.123 (a relative improvement of 10.9%) while slightly improving audio quality and speaker similarity, thereby significantly alleviating the quality–intelligibility trade-off.

Abstract

Generative streaming models for Target Speaker Extraction (TSE) commonly exhibit a quality--intelligibility trade-off, wherein naive optimization for perceptual audio quality tends to degrade speech intelligibility, and conversely. We reveal that this trade-off arises not from the constraints of streaming architectures, but from an inappropriate choice of optimization anchor. Directly optimizing against audio quality metrics induces catastrophic reward hacking, where content critical to pronunciation and intelligibility is systematically erased to maximize a proxy score. To break this bottleneck, we propose two complementary improvements: an enlarged Conformer convolution kernel for richer local spectro-temporal modeling, and WavLM-anchored Direct Preference Optimization (DPO) fine-tuning strategy. DPO preference pairs are ranked by WavLM cosine similarity, a deep acoustic feature encoding both phonetic structure and speaker identity, providing an optimization anchor that resists hacking. Under a 560 ms streaming chunk size, the proposed method achieves a 10.9% relative intelligibility improvement (word error rate: 0.138 to 0.123), with marginal simultaneous gains in audio quality and speaker similarity.

Problem

• Target Speaker Extraction
• quality--intelligibility trade-off
• streaming speech processing
• speech intelligibility
• audio quality

Innovation

• Target Speaker Extraction
• Direct Preference Optimization
• WavLM
• streaming speech processing
• quality-intelligibility trade-off