Beyond the Target: From Imitation to Collaboration in Speculative Decoding

2026-05-23

AI Summary

This work proposes Collaborative Speculative Decoding (CoSpec), a novel approach that addresses the limitation of traditional speculative decoding, which overly relies on the target model and overlooks the potential of the draft model to generate superior tokens at certain positions. CoSpec introduces, for the first time, a collaborative mechanism that departs from the autocratic paradigm of the target model by employing a reinforcement learning–trained arbitration policy. This policy dynamically selects, during discrepancies between draft and target model outputs, the token more likely to lead to the correct sequence. Integrated with parallel token verification and an efficient inference framework, CoSpec significantly accelerates large language model inference while outperforming baseline methods that rely solely on the target model.

Abstract

Speculative decoding (SPD) accelerates large language model (LLM) inference by letting a smaller draft model propose multiple future tokens that are verified in parallel by a larger target model. The dominant SPD paradigm treats the target model as the sole reliable teacher, accepting a draft token only when it exactly matches the target prediction. This design implicitly assumes that the target is always the better choice at every position. In practice, this assumption does not hold. Although the draft is the weaker model overall, it is not uniformly inferior at the token level. In a meaningful fraction of cases where draft and target disagree, the draft's choice is the one that leads to the correct final answer. Inspired by this, we introduce \textbf{Collaborative Speculative Decoding (CoSpec)}, a generalization of SPD that no longer treats the target model as the sole token-level authority. CoSpec trains an arbitration policy via reinforcement learning to decide whether to accept tokens from the draft or target model, selectively accepting draft tokens at mismatches when doing so is likely to yield a correct final answer. Experimental results show that CoSpec maintains substantial speedups while surpassing target-only performance. By shifting the emphasis from imitation to collaboration, CoSpec suggests a new perspective on speculative decoding.

Problem

• Speculative Decoding
• Large Language Models
• Token-level Disagreement
• Draft Model
• Target Model

Innovation

• Collaborative Speculative Decoding
• speculative decoding
• reinforcement learning
• arbitration policy
• large language model inference