Alignment Drift in Single-Model Speculative Decoding for ASR: Mechanism, Correction, and Cost

2026-08-12

AI Summary

This work addresses a critical limitation in single-model speculative decoding for automatic speech recognition, where the draft module struggles to accurately track audio positions, leading to alignment drift and degraded prediction quality. The study reveals that precise audio position tracking is pivotal for effective speculation and proposes AnchorDraft, a training method that corrects such drift without altering the inference graph. AnchorDraft either leverages attention readout positions during verification or guides the draft module to implicitly learn positional information. Experiments demonstrate that AnchorDraft significantly accelerates end-to-end inference across two target model scales. Properly aligned audio windows substantially increase token acceptance rates, with the median error in verification-stage attention confined to merely two frames.

Abstract

Speculative decoding speeds up generation by letting a cheap draft propose several tokens that a target model checks in one pass. In the single-model form, the draft is a lightweight module attached to the target rather than a separate model. Applying this design to Automatic Speech Recognition (ASR) introduces an extra problem. The draft can read the whole audio at every step, yet its proposals get worse as it runs on its own. Access is not localization. The accepted text keeps the transcript position explicit, but the draft must also track the changing audio position. In the primary matched comparison, per-step audio access changes the first proposal modestly but roughly doubles later-proposal acceptance. Fixed-width windows show that the audio position explains part of this gap. A correctly placed window recovers continuation, while an equally narrow window at the wrong position reduces it. Late-draft median error reaches 21 frames in the hardest reported condition, while target attention during verification stays within a 2-frame median. We test two ways to reduce this drift. The first reads the audio position from verification attention and uses it to guide the next draft round. It saves time only when the extra accepted tokens offset the readout cost. The second is AnchorDraft, which teaches the draft to track the audio position during training without changing the inference graph. The trained draft improves end-to-end speed at both tested target scales. These results show that ASR self-speculation depends on token prediction, audio-position tracking, and draft cost.

Problem

• speculative decoding
• alignment drift
• automatic speech recognition
• audio-position tracking
• single-model

Innovation

• speculative decoding
• alignment drift
• audio-position tracking
• AnchorDraft
• automatic speech recognition