Dynamic Clustering for Cross-Segment Permutation Alignment in Long Speech Separation

2026-08-10

AI Summary

This work addresses the challenge of inconsistent speaker permutation across segments in long-form speech separation, which arises from segment-wise processing. To resolve this without requiring additional training, the authors propose a dynamic clustering approach that maintains a continuously updated reference pool of speaker embeddings. By computing cosine similarities between embeddings from the current segment and those in the reference pool, the method predicts cross-segment speaker alignment and incrementally retains the most representative embeddings, enabling plug-and-play consistency. The approach demonstrates strong robustness under challenging conditions—such as unknown numbers of speakers and prolonged silent intervals—and significantly outperforms existing methods in both dense and sparse long-duration speech scenarios.

Abstract

Long speech separation typically employs a segment-separation-stitch paradigm where recordings are divided into short segments, processed independently, and stitched together. Its challenge lies in predicting cross-segment permutations. This paper proposes a training-free dynamic clustering approach for cross-segment permutation alignment using speaker embedding reference pools. The method predicts the permutation using the cosine similarity between current segment embeddings and the reference pools. The approach updates reference pools by retaining the most representative speaker embeddings based on their overall cosine similarity with existing references. As a plug-and-play post-processing module compatible with existing separation models, the proposed method demonstrates superior performance compared to existing methods on dense and sparse long speech scenarios, particularly in challenging sparse scenarios with extended utterance gaps, and further shows robustness to speaker count estimation errors in unknown speaker count scenarios.

Problem

• long speech separation
• cross-segment permutation
• speaker embedding
• permutation alignment
• segment stitching

Innovation

• dynamic clustering
• permutation alignment
• speaker embedding
• long speech separation
• plug-and-play post-processing