End-to-End Joint ASR and Speaker Role Diarization with Child-Adult Interactions

2026-01-25

AI Summary

This work addresses the cascading error problem in speaker role differentiation and speech recognition within child–adult conversations by proposing the first end-to-end joint modeling framework that unifies automatic speech recognition (ASR) and speaker role classification on top of the Whisper architecture. The approach integrates serialized label outputs, a lightweight frame-level role discrimination head, silence suppression, and state-machine-guided constrained decoding to enforce structural consistency and precise timestamp alignment, ensuring semantically and structurally valid outputs. Experimental results demonstrate that the proposed method significantly outperforms conventional cascaded systems on two datasets, achieving lower multi-speaker word error rates and competitive speaker role classification accuracy on both Whisper-small and Whisper-large variants.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Accurate transcription and speaker diarization of child-adult spoken interactions are crucial for developmental and clinical research. However, manual annotation is time-consuming and challenging to scale. Existing automated systems typically rely on cascaded speaker diarization and speech recognition pipelines, which can lead to error propagation. This paper presents a unified end-to-end framework that extends the Whisper encoder-decoder architecture to jointly model ASR and child-adult speaker role diarization. The proposed approach integrates: (i) a serialized output training scheme that emits speaker tags and start/end timestamps, (ii) a lightweight frame-level diarization head that enhances speaker-discriminative encoder representations, (iii) diarization-guided silence suppression for improved temporal precision, and (iv) a state-machine-based forced decoding procedure that guarantees structurally valid outputs. Comprehensive evaluations on two datasets demonstrate consistent and substantial improvements over two cascaded baselines, achieving lower multi-talker word error rates and demonstrating competitive diarization accuracy across both Whisper-small and Whisper-large models. These findings highlight the effectiveness and practical utility of the proposed joint modeling framework for generating reliable, speaker-attributed transcripts of child-adult interactions at scale. The code and model weights are publicly available

Problem

• speaker diarization
• automatic speech recognition
• child-adult interactions
• end-to-end modeling
• multi-talker transcription

Innovation

• end-to-end joint modeling
• speaker role diarization
• Whisper architecture
• serialized output training
• diarization-guided silence suppression