Depth Exploration for LLM Decoding

2026-06-28

AI Summary

This work addresses the computational redundancy in conventional autoregressive large language models, which process every token through all network layers despite many tokens becoming predictable at intermediate depths. To mitigate this inefficiency without altering generation outcomes, the authors propose Depth Exploration Decoding (DEX)—a method that parallelly explores multiple candidate exit depths while guaranteeing identical outputs to standard autoregressive decoding. DEX employs an “expand–commit–fold” pipeline, integrating early-exit architectures, final-depth verification, state reuse, and dynamic exploration lattice pruning to effectively exploit depth-wise redundancy. Experiments demonstrate that DEX consistently outperforms existing depth-adaptive approaches across various large language models, achieving end-to-end throughput comparable to speculative and distributed decoding, with performance further improving as exploration granularity increases.

Abstract

Autoregressive LLM decoding evaluates every generated token through the full layer stack, even though many tokens become predictable at intermediate depths. Existing lossless depth-adaptive methods exploit this redundancy by choosing a single non-final exit depth and verifying its prediction with the final-depth model. However, our measurements show that this selection-based strategy leaves substantial headroom: choosing an exit too late wastes computation, while choosing one too early triggers fallback and discards dependent drafts. We propose Depth Exploration Decoding (DEX), a lossless decoding algorithm that replaces single-depth selection with parallel exploration over multiple candidate depths. At each commit position, DEX validates candidates against the final-depth reference, commits exactly the final-depth token, and collapses the exploration lattice to retain only reusable branch states. This expand--commit--collapse procedure preserves equivalence to standard autoregressive decoding while reducing the cost of committing each token. Across early-exit-trained and standard LLMs, DEX outperforms representative depth-selection baselines and achieves competitive end-to-end throughput against speculative and distributed decoding methods. Moreover, DEX improves as the explored depths become finer, showing that parallel depth exploration provides a scalable way to exploit the underused depth axis of LLM decoding.

Problem

• LLM decoding
• depth-adaptive
• autoregressive generation
• computational redundancy
• early exit

Innovation

• depth-adaptive decoding
• parallel depth exploration
• lossless acceleration
• early exit
• autoregressive LLM decoding