OUTLETS: Output-Length Prediction from Speculative Decoding Backbones

2026-09-01

AI Summary

针对大型语言模型输出长度预测问题，提出OUTLETS方法，利用推测解码的潜在表示作为长度预测器，减少延迟并提高调度效率。

Abstract

The heavy-tailed distribution of output lengths in Large Language Model (LLM) serving poses major challenges for resource provisioning and cluster scheduling. Although output-length prediction can mitigate these issues, existing approaches have key drawbacks: external proxy models add substantial latency and often have limited fidelity, whereas internal state-based methods are efficient but rely on shallow probes of current model states. We identify a structural connection between speculative decoding (SD) and length prediction: latent representations produced by the draft decoder in advanced frameworks (e.g., EAGLE-3) encode signals that are predictive of generation length. Building on this insight, we introduce OUTLETS (Output-Length Prediction from Speculative Decoding Backbones), which repurposes the speculative backbone as a trajectory-aware length predictor. When its draft representations are already computed for speculative decoding, OUTLETS adds only a lightweight regression head and achieves lower MAE than the evaluated methods. Under saturated disaggregated serving, OUTLETS predictions enable standard scheduling policies to prioritize shorter requests and distribute requests more evenly across decoding instances, reducing short-request P99 latency by 34.8%.

Problem

• Large Language Model
• output-length prediction
• resource provisioning
• cluster scheduling

Innovation

• speculative decoding
• output-length prediction
• trajectory-aware length predictor
• lightweight regression head