Position-Aware Target Speaker Extraction for Long-Form Multi-Party Conversations: A Diarization-Free Framework for ASR

2026-06-28

AI Summary

This work addresses the challenge of accurately determining “who spoke when” in long-form multi-speaker conversations, where speaker activity is sparse and speech overlaps frequently. The authors propose PATSE, a novel framework that, for the first time, incorporates direction-of-arrival (DOA) as a spatial prior into target speaker extraction. By leveraging a multi-channel, location-aware mechanism, PATSE directly generates speaker-attributed speech streams and employs lightweight post-processing to infer speaker activity—eliminating the need for explicit speaker diarization. This end-to-end approach effectively mitigates cross-window speaker inconsistency and crosstalk. Experimental results demonstrate that PATSE significantly outperforms conventional continuous speech separation methods and diarization-based pipeline systems on both simulated and real meeting data, consistently yielding improved automatic speech recognition performance.

Abstract

In long-form multi-party conversations, highly imbalanced speaker activity and frequent overlap make it difficult to identify "who spoke when and what". Sliding-window continuous speech separation (CSS) mitigates sparse supervision, but often suffers from cross-window speaker inconsistency and residual crosstalk, which in practice requires diarization for reliable speaker attribution. Motivated by the stability of speakers' directions of arrival (DOAs) in meetings, we propose PATSE, a multi-channel Position-Aware Target Speaker Extraction front-end that uses DOA as a spatial prior to directly extract the speech of each target speaker. PATSE combines a DOA-guided spatial encoder and conditioner to generate speaker-attributed streams, from which speaker activity can be inferred via simple post-processing (e.g., VAD) without explicit diarization. Experiments on both replayed and real conversations show consistent ASR gains outperforming CSS and diarization-based pipelines.

Problem

• multi-party conversations
• speaker extraction
• speaker diarization
• speech separation
• direction of arrival

Innovation

• Position-Aware Target Speaker Extraction
• Direction of Arrival (DOA)
• Diarization-Free
• Multi-Channel Speech Separation
• Long-Form Multi-Party Conversations