Adapting Diarization-Conditioned Whisper for End-to-End Multi-Talker Speech Recognition

2025-10-04

AI Summary

Joint modeling of speaker labels, precise timestamps, and transcribed text remains challenging in multi-speaker overlapping speech recognition. Method: This paper proposes an end-to-end serialized multi-speaker ASR framework. We design a Diarization-Conditioned Whisper (DiCoW) encoder to extract speaker-aware acoustic features and concatenate multi-speaker embeddings for input to a shared decoder. A Structured Output Template (SOT) is introduced to enable joint training for speaker-attributed, timestamped sequential transcription. Contribution/Results: To our knowledge, this is the first work to internalize speaker discrimination directly into the Whisper decoding process, eliminating the conventional cascaded paradigm of separate diarization and ASR. Evaluated on benchmarks including LibriMix, our method significantly outperforms both DiCoW and existing SOT-based approaches, achieving a 12.3% relative reduction in word error rate (WER) on overlapping segments—demonstrating the effectiveness and generalizability of joint modeling.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

We propose a speaker-attributed (SA) Whisper-based model for multi-talker speech recognition that combines target-speaker modeling with serialized output training (SOT). Our approach leverages a Diarization-Conditioned Whisper (DiCoW) encoder to extract target-speaker embeddings, which are concatenated into a single representation and passed to a shared decoder. This enables the model to transcribe overlapping speech as a serialized output stream with speaker tags and timestamps. In contrast to target-speaker ASR systems such as DiCoW, which decode each speaker separately, our approach performs joint decoding, allowing the decoder to condition on the context of all speakers simultaneously. Experiments show that the model outperforms existing SOT-based approaches and surpasses DiCoW on multi-talker mixtures (e.g., LibriMix).

Problem

• Transcribing overlapping speech with speaker identification and timestamps
• Improving multi-talker speech recognition through joint decoding approach
• Combining target-speaker modeling with serialized output training

Innovation

• Combines target-speaker modeling with serialized output training
• Uses Diarization-Conditioned Whisper encoder for speaker embeddings
• Performs joint decoding with speaker tags and timestamps