ECHO: Early-layer Collaborative Hierarchical Orchestration with Bonus Logits in Speculative Decoding

2026-09-15

AI Summary

为解决草稿模型自由推测解码中的陈旧候选和高验证成本问题，提出ECHO框架，利用早期层高效区分与最终层权威分布的特性，通过内外双循环提高效率。

Abstract

While draft-model-free speculative decoding offers a promising path to efficient LLM inference, it is frequently constrained by stale draft candidates and the high computational cost of the verification. To address these challenges, we propose ECHO, a hierarchical dual-loop framework that exploits the functional asymmetry between LLM layers. Leveraging the high discriminative efficiency of early layers and the authoritative distribution of final layers, ECHO bifurcates inference into a high-frequency inner loop and a low-frequency outer loop. Within the inner loop, early-layer bonus logits drive rapid, multi-step draft-tree exploration at a minimal cost. Simultaneously, the outer loop performs authoritative full-model verification through a state-reuse mechanism. Crucially, the outer loop also utilizes final-layer bonus logits to correct existing paths and supplement the tree with high-confidence candidates for subsequent cycles. Experimental results across diverse benchmarks demonstrate that ECHO significantly boosts mean accepted tokens and achieves a 2.4$\times$ to 2.9$\times$ speedup, outperforming existing state-of-the-art baselines with negligible engineering overhead and no extra deployment parameters, albeit with a one-shot fine-tuning dependency for optimal acceleration. The code is available at https://github.com/whucs21Mzy/ECHO.

Problem

• speculative decoding
• LLM inference
• stale draft candidates
• high computational cost

Innovation

• speculative decoding
• hierarchical dual-loop framework
• bonus logits
• state-reuse mechanism
• early-layer and final-layer collaboration