G-STAR: End-to-End Global Speaker-Tracking Attributed Recognition

2026-03-11

AI Summary

This work addresses the challenges of speaker identity inconsistency and imprecise timestamps arising from chunked processing in long-form, multi-talker overlapping speech. To this end, we propose G-STAR, an end-to-end system that jointly models a global time-aware speaker tracking module with a speech large language model for the first time. G-STAR leverages structured, time-anchored speaker cues to guide conditional text generation and employs a hierarchical objective optimization strategy to balance local accuracy with long-range contextual consistency under heterogeneous supervision and domain shift scenarios. Experimental results demonstrate that G-STAR significantly outperforms existing methods in both speaker attribution accuracy and temporal boundary precision.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We study timestamped speaker-attributed ASR for long-form, multi-party speech with overlap, where chunk-wise inference must preserve meeting-level speaker identity consistency while producing time-stamped, speaker-labeled transcripts. Previous Speech-LLM systems tend to prioritize either local diarization or global labeling, but often lack the ability to capture fine-grained temporal boundaries or robust cross-chunk identity linking. We propose G-STAR, an end-to-end system that couples a time-aware speaker-tracking module with a Speech-LLM transcription backbone. The tracker provides structured speaker cues with temporal grounding, and the LLM generates attributed text conditioned on these cues. G-STAR supports both component-wise optimization and joint end-to-end training, enabling flexible learning under heterogeneous supervision and domain shift. Experiments analyze cue fusion, local versus long-context trade-offs and hierarchical objectives.

Problem

• speaker-attributed ASR
• multi-party speech
• speaker diarization
• time-stamped transcription
• speaker identity consistency

Innovation

• speaker-attributed ASR
• end-to-end system
• time-aware speaker tracking
• Speech-LLM
• global identity consistency