Streaming Sortformer: Speaker Cache-Based Online Speaker Diarization with Arrival-Time Ordering

2025-07-24

AI Summary

This work addresses two key challenges in online speaker diarization: (1) misalignment between output speaker order and their actual temporal onset, and (2) high latency in multi-speaker tracking. To this end, we propose a streaming speaker diarization method based on arrival-time ordering. Our core innovation is a speaker embedding caching mechanism that dynamically maintains an ordered speaker representation list indexed by first-occurrence time; it jointly leverages frame-level acoustic embeddings and model prediction scores to drive adaptive cache updates. Furthermore, we introduce Sortformer—a transformer-based architecture—to explicitly model sequential dependencies among speakers. Evaluated on standard benchmarks under strict low-latency constraints, our approach achieves significant improvements in both temporal ordering consistency and tracking accuracy, while preserving real-time processing capability and robustness. The proposed framework offers a scalable, deployment-friendly paradigm for streaming multi-speaker analysis.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Natural Language Processing: Speech

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

This paper presents a streaming extension for the Sortformer speaker diarization framework, whose key property is the arrival-time ordering of output speakers. The proposed approach employs an Arrival-Order Speaker Cache (AOSC) to store frame-level acoustic embeddings of previously observed speakers. Unlike conventional speaker-tracing buffers, AOSC orders embeddings by speaker index corresponding to their arrival time order, and is dynamically updated by selecting frames with the highest scores based on the model's past predictions. Notably, the number of stored embeddings per speaker is determined dynamically by the update mechanism, ensuring efficient cache utilization and precise speaker tracking. Experiments on benchmark datasets confirm the effectiveness and flexibility of our approach, even in low-latency setups. These results establish Streaming Sortformer as a robust solution for real-time multi-speaker tracking and a foundation for streaming multi-talker speech processing.

Problem

• Develops real-time speaker diarization with arrival-time ordering
• Dynamically manages speaker embeddings for efficient cache usage
• Enables low-latency multi-speaker tracking in streaming scenarios

Innovation

• Arrival-Order Speaker Cache for dynamic embedding storage
• Speaker index ordering by arrival time
• Dynamic update mechanism for efficient cache usage