Discriminative-Generative Target Speaker Extraction with Decoder-Only Language Models

2026-01-09

AI Summary

This work addresses the common trade-off in target speaker extraction, where aggressive suppression of interfering speakers often degrades speech perceptual quality and naturalness. To overcome this limitation, the authors propose a hybrid discriminative–generative framework: a discriminative front-end robustly extracts target speaker embeddings, while a novel decoder-only language model—introduced for the first time in this context—reconstructs high-fidelity speech in a neural audio codec space, either autoregressively or non-autoregressively. Through a two-stage architecture and a collaborative training strategy, the method effectively balances speech quality, intelligibility, and speaker consistency beyond conventional SI-SDR optimization, significantly enhancing both the naturalness of the generated speech and the overall system robustness.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Target speaker extraction (TSE) aims to recover the speech signal of a desired speaker from a mixed audio recording, given a short enrollment utterance. Most existing TSE approaches are based on discriminative modeling paradigms. Although effective at suppressing interfering speakers, these methods often struggle to produce speech with high perceptual quality and naturalness. To address this limitation, we first propose LauraTSE, a generative TSE model built upon an auto-regressive decoder-only language model. However, purely generative approaches may suffer from hallucinations, content drift, and limited controllability, which may undermine their reliability in complex acoustic scenarios. To overcome these challenges, we further introduce a discriminative-generative TSE framework. In this framework, a discriminative front-end is employed to robustly extract the target speaker's speech, yielding stable and controllable intermediate representations. A generative back-end then operates in the neural audio codec representation space to reconstruct fine-grained speech details and enhance perceptual quality. This two-stage design effectively combines the robustness and controllability of discriminative models with the superior naturalness and quality enhancement capabilities of generative models. Moreover, we systematically investigate collaborative training strategies for the proposed framework, including freezing or fine-tuning the front-end, incorporating an auxiliary SI-SDR loss, and exploring both auto-regressive and non-auto-regressive inference mechanisms. Experimental results demonstrate that the proposed framework achieves a more favorable trade-off among speech quality, intelligibility, and speaker consistency.

Problem

• Target Speaker Extraction
• Speech Quality
• Generative Models
• Discriminative Models
• Perceptual Naturalness

Innovation

• target speaker extraction
• discriminative-generative framework
• decoder-only language model
• neural audio codec
• collaborative training