Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting

2026-07-12

AI Summary

This work addresses the limitations of conventional speculative decoding, which relies on auxiliary modules that incur additional training and communication overhead and fail to fully exploit the inherent parallelism of large language models. The authors propose Progressive Tree-based Drafting (PTD), a model-agnostic speculative decoding framework that requires neither extra training nor architectural modifications. PTD enables structurally coordinated parallel draft generation within the target model itself by leveraging a progressive tree structure combined with iterative pruning, allowing multiple semantic paths to be explored in a single forward pass while balancing diversity and coherence. Evaluated across multiple benchmarks, PTD achieves up to 2× inference speedup over baseline methods without compromising generation quality.

Abstract

Speculative decoding has significantly accelerated Large Language Model (LLM) inference by alleviating memory-bound bottlenecks. However, traditional speculative decoding typically relies on auxiliary draft modules, incurring significant training and communication overhead. Although recent methods attempt to generate drafts within the target model itself, they often fail to fully exploit its latent parallel capacity due to a lack of structural coordination. In this paper, we propose \textbf{Progressive Tree Drafting (PTD)}, which employs a structured, guided parallel drafting strategy to harness the model's parallel potential. By coupling a progressive tree structure with a stepwise pruning mechanism, PTD actively guides the LLM to explore multiple semantic paths in a single forward pass, ensuring both draft diversity and coherence. Experiments demonstrate that PTD achieves up to $2\times$ decoding speedup across various benchmarks while remaining training-free and model-agnostic. Our code is available at: https://github.com/MINE-USTC/PTD.

Problem

• speculative decoding
• autoregressive language models
• parallelism
• draft generation
• inference acceleration

Innovation

• speculative decoding
• progressive tree drafting
• parallel inference
• training-free acceleration
• model-agnostic