🤖 AI Summary
This study investigates how cognitive task types influence token acceptance probability in tree-based speculative decoding. Leveraging TinyLlama-1.1B as the draft model and Llama-2-7B-Chat-GPTQ as the target model with tree attention, the authors analyze acceptance dynamics across 99,768 speculative nodes spanning four task categories: code generation, mathematical reasoning, logical reasoning, and open-ended dialogue. The work reveals, for the first time, that task type is a stronger predictor of acceptance probability than tree depth. Notably, despite exhibiting the highest entropy, chat tasks achieve the highest acceptance rate (mean accepted length > 1.0), attributed to the stylistic predictability induced by RLHF alignment. A weak negative correlation between domain entropy and acceptance rate (ρ ∈ [−0.20, −0.15]) further provides empirical grounding for domain-adaptive speculative budget allocation.
📝 Abstract
Speculative decoding accelerates large language model (LLM) inference. It uses a small draft model to propose a tree of future tokens. A larger target model then verifies these tokens in a single batched forward pass. Despite the growing body of work on speculative methods, the degree to which the cognitive characteristics of a task affect acceptance probability remains largely unexplored. We present an empirical study of tree-based speculative decoding acceptance dynamics. Our study spans four well-established NLP benchmark domains: code generation, mathematical reasoning, logical reasoning, and open-ended chat. For this, we use TinyLlama-1.1B as the draft model against Llama-2-7B-Chat-GPTQ as the target. Over 99,768 speculative nodes collected from 200 prompts, we derive per-domain acceptance rates, expected accepted lengths, depth-acceptance profiles, and entropy-acceptance correlations. We find that task type is a stronger predictor of acceptance than tree depth. Furthermore, only the chat domain consistently yields an expected accepted length exceeding 1.0 token per step. We also show that the entropy-acceptance correlation is consistently negative but weak across all domains (rho in [-0.20, -0.15]). Counterintuitively, chat produces the highest entropy yet the highest acceptance rate. We attribute this divergence to the lexical predictability of RLHF-aligned register. These findings have direct implications for domain-aware speculation budgets and draft-model selection strategies. Index Terms--speculative decoding, large language model inference, tree attention, draft model, acceptance probability, LLM efficiency