Lightweight Target-Speaker-Based Overlap Transcription for Practical Streaming ASR

2025-06-25

AI Summary

Overlapping speech severely degrades ASR performance in dynamic multi-speaker scenarios such as broadcast and televised debates. To address this, we propose a lightweight, target-speaker-oriented streaming ASR extension framework. Our approach features three key contributions: (1) a decoupled architecture that reuses a speaker-agnostic ASR backbone while introducing a lightweight, frozen-feature-trained binary classifier to dynamically trigger overlap-aware processing; (2) a Feature-wise Linear Modulation (FiLM) mechanism that injects speaker embeddings into the ASR decoder at low computational cost, enabling selective transcription; and (3) an end-to-end joint training strategy using synthetic overlapped mixtures. Evaluated on Czech televised debate data containing 16% overlapping speech, our method reduces WER on overlapping segments from 68.0% to 35.78%, with only a 44% increase in overall computational overhead—achieving both high accuracy and strong scalability.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Overlapping speech remains a major challenge for automatic speech recognition (ASR) in real-world applications, particularly in broadcast media with dynamic, multi-speaker interactions. We propose a light-weight, target-speaker-based extension to an existing streaming ASR system to enable practical transcription of overlapping speech with minimal computational overhead. Our approach combines a speaker-independent (SI) model for standard operation with a speaker-conditioned (SC) model selectively applied in overlapping scenarios. Overlap detection is achieved using a compact binary classifier trained on frozen SI model output, offering accurate segmentation at negligible cost. The SC model employs Feature-wise Linear Modulation (FiLM) to incorporate speaker embeddings and is trained on synthetically mixed data to transcribe only the target speaker. Our method supports dynamic speaker tracking and reuses existing modules with minimal modifications. Evaluated on a challenging set of Czech television debates with 16% overlap, the system reduced WER on overlapping segments from 68.0% (baseline) to 35.78% while increasing total computational load by only 44%. The proposed system offers an effective and scalable solution for overlap transcription in continuous ASR services.

Problem

• Overlapping speech challenges real-world ASR systems
• Lightweight solution needed for streaming ASR with minimal overhead
• Target-speaker transcription in multi-speaker broadcast scenarios

Innovation

• Lightweight target-speaker-based overlap transcription
• Compact binary classifier for overlap detection
• Feature-wise Linear Modulation for speaker conditioning