EvoSpec: Evolving Speculative Decoding via Real-Time Vocabulary and Parameter AdaptationTarget

2026-04-17

AI Summary

This work addresses the limitations of existing static pruning methods in speculative decoding for large language models, which struggle to adapt to domain or topic shifts—leading to reduced acceptance rates—and suffer from performance bottlenecks in the output projection layer as vocabulary size grows. To overcome these challenges, the authors propose the first speculative decoding framework capable of real-time evolution, featuring context-aware retrieval of critical long-tail tokens, dynamic vocabulary pruning via a hybrid semantic-statistical index, and a lightweight online curriculum learning alignment mechanism. This enables continuous online adaptation of the draft model, progressively narrowing its distributional gap with the target model. Evaluated on specialized domains such as code, legal, and medical texts, the method achieves a 1.13× speedup over the FR-Spec baseline while reducing memory overhead by 27%, substantially outperforming current static approaches.

Abstract

Speculative decoding accelerates Large Language Model inference via a draft-then-verify paradigm, yet the output projection layer becomes a bottleneck as vocabulary sizes scale. While existing static pruning methods effectively reduce this overhead, they suffer from precipitous drops in acceptance rate in specialized domains or topic-switching scenarios due to their inability to capture dynamic distribution shifts. To address this, we introduce EvoSpec, a framework that enables real-time evolution of the draft model through dynamic vocabulary and parameter adaptation. Unlike static or purely retrieval-based approaches, EvoSpec employs a context-aware mechanism that retrieves critical long-tail tokens via efficient semantic and statistical indexing. Furthermore, we propose a lightweight online alignment strategy utilizing curriculum learning to continually minimize the distributional gap between the draft and target models. Extensive evaluations across specialized domains (coding, law, and medicine) confirm that EvoSpec overcomes the limitations of static baselines. On EAGLE-3, it achieves a 1.13x speedup in these settings over the state-of-the-art static baseline FR-Spec, with 27\% lower memory overhead than standard online adaptation.

Problem

• speculative decoding
• vocabulary scaling
• distribution shift
• acceptance rate
• large language models

Innovation

• speculative decoding
• dynamic vocabulary adaptation
• context-aware retrieval
• online alignment
• curriculum learning