CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration

2026-05-11

AI Summary

This work addresses the memory bandwidth bottleneck that limits autoregressive decoding of large language models on memory-constrained devices, where existing speculative decoding approaches struggle due to the simultaneous loading of both target and draft models. To overcome this challenge, the authors propose a cascaded self-speculative decoding framework that operates with only the target model, integrating parameter offloading, memory-aware scheduling, and a tree-structured verification mechanism to enable efficient self-speculation without any auxiliary draft model. Evaluated on real edge devices, the method achieves up to 5.08× end-to-end speedup with no degradation in generation quality, outperforming the current state-of-the-art by 1.45× while significantly reducing peak memory usage and improving token acceptance rates.

Abstract

Auto-regressive decoding in Large Language Models (LLMs) is inherently memory-bound: every generation step requires loading the model weights and intermediate results from memory (e.g., High-Bandwidth Memory (HBM) for GPU servers), making throughput bottlenecked by memory bandwidth rather than compute. Speculative decoding addresses this by enabling parallel verification of multiple draft tokens, effectively amortizing the cost of each target-model call. However, existing speculative decoding methods are designed under the assumption that HBM is sufficiently large to hold both the target model and an auxiliary draft model simultaneously -- an assumption that breaks down on memory-constrained devices such as edge platforms with limited DRAM. We analyze the inference bottleneck in this memory-limited regime and propose CATS, a self-speculative decoding framework that conducts cascaded verification and correction based on the memory budget and parameter offloading patterns on memory-limited devices. This design maximizes token acceptance rate and end-to-end speedup while keeping the peak memory footprint on the device equal to that of the target model alone. We evaluate CATS on different models across five benchmarks on real edge devices. CATS can achieve a wall-clock speedup of up to 5.08x with no degradation in generation quality, outperforming the SOTA method by up to 1.45x under edge memory constraints.

Problem

• speculative decoding
• memory-constrained devices
• LLM inference
• memory bandwidth bottleneck
• edge platforms

Innovation

• speculative decoding
• memory-constrained inference
• edge LLM acceleration
• cascaded verification
• parameter offloading