PS4: Proxy-Supervised Joint Training for Real Target Speaker Extraction

2026-07-09

AI Summary

This work addresses the challenges of target speaker extraction in real-world conversational scenarios, where large-scale training data and clean supervisory signals are typically unavailable. To this end, the authors construct a training corpus comprising 71,771 multilingual speech mixtures and propose the first proxy-supervised joint training framework tailored for realistic dialogues. Built upon a BSRNN-based separator, the framework enables end-to-end optimization by integrating four differentiable objectives: ASR cross-entropy loss, speaker embedding similarity, frame-level voice activity detection, and perceptual speech quality. Without access to clean reference utterances during training, the proposed method achieves substantial performance gains, securing second place overall in the REAL-T Challenge while attaining the highest scores in speaker similarity and temporal alignment F1 metrics.

Abstract

Training target speaker extraction (TSE) models for real conversational mixtures remains challenging because large-scale training corpora and clean target speech for supervision are unavailable. We present PS4, a proxy-supervised training framework for TSE in real conversational mixtures, with two main contributions. First, we construct a large-scale corpus of 71,771 training samples derived from four public datasets, covering both Chinese and English scenarios. Each sample contains an overlapping speech mixture, per-speaker enrollment audio, a ground-truth transcript, and frame-level voice activity labels. Second, we propose a proxy-supervised joint training strategy that fine-tunes a BSRNN-based TSE model using four complementary differentiable objectives: ASR cross-entropy, speaker similarity, frame-level voice activity detection, and perceptual audio quality. Starting from a publicly available pre-trained checkpoint, only the BSRNN separator is updated during fine-tuning. On the REAL-T challenge leaderboard, PS4 ranks 2nd overall, achieving the best speaker similarity and timing F1 among all submitted systems.

Problem

• target speaker extraction
• real conversational mixtures
• supervision
• training data scarcity
• overlapping speech

Innovation

• proxy-supervised training
• target speaker extraction
• joint optimization
• real conversational mixtures
• BSRNN