REDDIT: Correcting Model-Generated Timestamp Drift in ASR without Forgetting via Replay-Based Distribution Editing

2026-07-06

AI Summary

This work addresses timestamp drift in autoregressive automatic speech recognition (ASR) systems during long non-speech intervals and the catastrophic forgetting induced by conventional fine-tuning. The authors propose REDDIT, a two-stage replay-based distribution editing framework that enables unsupervised timestamp correction. First, timestamp targets are edited within the model’s own decoding context while preserving the distribution of non-timestamp tokens; second, a short-prefix fine-tuning step is applied. Supervision signals are derived solely from voice activity detection (VAD)-based speech segment cropping and offset stitching, eliminating the need for manual annotations. Evaluated on Whisper-tiny with only 1.6% of parameters updated and 34.9 hours of data, the method improves mean Intersection-over-Union (mIoU) for long gaps from 38.7% to 95.0%, reduces out-of-domain mixed-gap average alignment shift (AAS) from 2752 ms to 223 ms, and maintains a CommonVoice English (CV-en) match error rate (MER) of 41.3%.

Abstract

Modern autoregressive ASR systems can emit timestamps as decoded tokens, enabling timestamped transcription without frame-level aligners or inference-time post-processing. We show that these generated timestamps can drift across long non-speech spans: the transcript may remain plausible, but the decoded time axis drifts away from the audio. We study this non-speech-induced timestamp drift with self-built gap and long-gap benchmarks across 15 evaluated timestamp-producing ASR and audio-language systems. Naive timestamp-corrected fine-tuning improves alignment but can severely degrade non-target ASR behavior, exposing a forgetting problem. We propose REDDIT(REplay-based Distribution eDITing), a lightweight two-stage post-training framework that corrects timestamps while avoiding this catastrophic forgetting: it first edits timestamp targets under the model's own replayed decoder context while matching the frozen base distribution on non-timestamp tokens, then applies a short edited-prefix refinement stage. In this framework, we construct correction supervision without human transcripts or human timestamp annotations by combining VAD-trimmed speech spans with inserted non-speech gaps and known concatenation offsets. On Whisper-tiny, 34.9 hours of targeted correction audio used and only 1.6% of model parameters updated, raising long-gap mIoU from 38.7% to 95.0% and reducing mixed-gap out-of-domain AAS from 2752 ms to 223 ms while preserving CV-en MER at 41.3% (versus 524.2% for ordinary SFT decoder tuning).

Problem

• timestamp drift
• automatic speech recognition
• catastrophic forgetting
• non-speech segments
• temporal alignment

Innovation

• timestamp drift correction
• catastrophic forgetting
• replay-based distribution editing
• autoregressive ASR
• unsupervised timestamp supervision