Learning to Hear Hesitation: Continual Learning for Disfluency-Aware ASR

2026-06-12

AI Summary

This study addresses the poor performance of current automatic speech recognition (ASR) systems on disfluent speech—characterized by hesitations, repetitions, and other non-fluencies—which often leads to information loss or hallucination due to the omission of disfluent elements. The work proposes a novel approach that explicitly incorporates disfluency tags into a pre-trained ASR model and combines them with continual learning to enable incremental adaptation across diverse disfluency distributions while mitigating catastrophic forgetting. Experimental results demonstrate that the method significantly enhances robustness on multiple disfluent speech datasets without compromising overall recognition accuracy. Furthermore, the study uncovers a trade-off between learning disfluency markers and recognition performance and identifies a stable cross-attention head mechanism shared across methods, offering new insights into the internal dynamics of ASR models.

Abstract

Despite advances in large-scale Automatic Speech Recognition (ASR), disfluent speech remains challenging, as state-of-the-art systems are often optimized to omit disfluencies, leading to information loss and hallucinations. Prior work has focused on verbatim transcription and the integration of disfluency markers, but adapting models on limited datasets can lead to catastrophic forgetting of general-domain knowledge. We address this gap by leveraging continual learning (CL) with explicit disfluency tokens. We first introduce these tokens into a pretrained ASR model to establish stable token mechanisms, and then continue training on additional datasets with varying disfluency distributions. Through a detailed analysis of model dynamics during training, we identify a trade-off between marker learning and ASR performance, and a consistent cross-attention head mechanism shared across CL methods.

Problem

• disfluency
• automatic speech recognition
• catastrophic forgetting
• continual learning
• hesitation

Innovation

• continual learning
• disfluency-aware ASR
• disfluency tokens
• cross-attention mechanism
• catastrophic forgetting