H-SAGE: Holistic Speaker-Aware Guided Experts for MoE-based Multi-Talker ASR

2026-07-01

AI Summary

This work addresses the challenge of inaccurate transcription in multi-talker speech recognition under high-overlap conditions, where existing Mixture-of-Experts (MoE) approaches suffer from temporal myopia due to frame-level routing and insufficiently supervised representation learning. To overcome these limitations, the authors propose H-SAGE, a novel framework that integrates acoustic state guidance into the MoE architecture for the first time. H-SAGE introduces a speaker-aware global encoder and a global-local joint gating mechanism, complemented by an explicit overlap-aware loss to enhance expert collaboration and contextual modeling. Evaluated on the LibriSpeechMix dataset, H-SAGE substantially outperforms strong baselines, demonstrating particularly robust performance in complex, highly overlapping scenarios.

Abstract

Multi-talker Automatic Speech Recognition (MTASR) faces significant challenges in accurately transcribing overlapping speech, particularly under complex high-overlap conditions. While recent Mixture-of-Experts (MoE) approaches have shown promise, they typically rely on frame-independent routing that leads to temporal myopia, and depend solely on the downstream ASR objective, which results in implicit and ungrounded representation learning. To address these limitations, we propose Holistic Speaker-Aware Guided Experts (H-SAGE) for MoE-based MTASR. Specifically, we introduce a Speaker-Aware Global Encoder to capture long-term dependencies, supervised by an auxiliary Overlap-Aware Loss that explicitly guides the model to discern acoustic states. Furthermore, we design a Holistic Gating Mechanism to arbitrate expert selection by jointly evaluating global context and local details. Experiments on LibriSpeechMix demonstrate that H-SAGE achieves consistent improvements over strong baselines, particularly in complex scenarios, validating that explicit acoustic guidance effectively enhances expert collaboration. Our code can be found at https://github.com/NKU-HLT/H-SAGE.

Problem

• Multi-talker ASR
• overlapping speech
• Mixture-of-Experts
• temporal myopia
• representation learning

Innovation

• Mixture-of-Experts
• Multi-talker ASR
• Speaker-Aware Encoding
• Holistic Gating
• Overlap-Aware Loss