🤖 AI Summary
This work addresses the computational bottleneck in speculative decoding caused by the full-vocabulary LM-head of draft models. The authors propose a low-rank parameterized LM-head architecture that compresses internal representations while preserving support for the complete vocabulary, without requiring vocabulary pruning or substantial modifications to the training pipeline. Built upon low-rank matrix decomposition and integrated into the EAGLE-3 framework, the method is validated across diverse target models and benchmarks. Experimental results demonstrate that, compared to the standard LM-head, the proposed approach achieves a 4–5× speedup in draft token generation and an 8–9% improvement in end-to-end inference throughput, while remaining compatible with various draft model architectures.
📝 Abstract
Speculative decoding speeds up autoregressive generation in Large Language Models (LLMs) through a two-step procedure, where a lightweight draft model proposes tokens which the target model then verifies in a single forward pass. Although the drafter network is small in modern architectures, its LM-head still performs projection to a large vocabulary, becoming one of the major computational bottlenecks. In prior work this issue has been predominantly addressed via static or dynamic vocabulary truncation. Yet mitigating the bottleneck, these methods bring in extra complexity, such as special vocabulary curation, sophisticated inference-time logic or modifications of the training setup. In this paper, we propose SlimSpec, a low-rank parameterization of the drafter's LM-head that compresses the inner representation rather than the output, preserving full vocabulary support. We evaluate our method with EAGLE-3 drafter across three target models and diverse benchmarks in both latency- and throughput-bound inference regimes. SlimSpec achieves $4\text{-}5\times$ acceleration over the standard LM-head architecture while maintaining a competitive acceptance length, surpassing existing methods by up to $8\text{-}9\%$ of the end-to-end speedup. Our method requires minimal adjustments of training and inference pipelines. Combined with the aforementioned speedup improvements, it makes SlimSpec a strong alternative across wide variety of draft LM-head architectures.