Post-Training Speech Enhancement Language Models with Perceptual Rewards

2026-06-19

AI Summary

This work addresses the misalignment between token-level cross-entropy loss used in existing speech enhancement language models and perceptual quality metrics such as DNSMOS, WER, and UTMOS. To bridge this gap, the authors propose a post-training stage for autoregressive speech enhancement models (UniSE/GenSE) that directly optimizes multiple non-differentiable perceptual scores as reward signals through multi-objective reinforcement learning. They introduce Grouped Sequence Policy Optimization (GSPO), which avoids reward gaming inherent in single-metric optimization and operates without surrogate models or offline preference data. Evaluated on the DNS2020 benchmark, the method achieves state-of-the-art performance, with human evaluations confirming its significant superiority over variants optimized for individual metrics.

Abstract

Speech enhancement language models achieve strong results when trained on discrete audio tokens, but their optimization relies on token-level cross-entropy rather than the perceptual metrics used for evaluation. We introduce a post-training stage for autoregressive speech enhancement language models using Group Sequence Policy Optimization (GSPO) with multi-metric perceptual rewards. Our method directly optimizes non-differentiable quality metrics (DNSMOS, WER, and UTMOS) as reward signals, without learned surrogates or offline preference pairs. Applied to two autoregressive base models, UniSE and GenSE, our approach achieves state-of-the-art results on the DNS2020 benchmark. A human evaluation ablation further shows that the composite multi-metric reward is preferred over any single-metric variant, confirming that multi-reward optimization avoids the reward hacking observed with single-metric training.

Problem

• speech enhancement
• perceptual metrics
• post-training
• reward optimization
• language models

Innovation

• speech enhancement
• perceptual rewards
• policy optimization
• multi-metric optimization
• post-training