🤖 AI Summary
This work addresses the high prefill overhead incurred by long-context prompts in large language model (LLM) agents, a challenge exacerbated by existing compression methods that rely on same-family small draft models. To overcome this limitation, the authors propose a training-free, cross-model-family prompt compression approach that leverages heterogeneous small draft models—such as Qwen, LLaMA, and DeepSeek—to evaluate token importance via attention mechanisms and guide speculative prefilling. This method is the first to demonstrate the effectiveness of cross-family draft models for prompt compression, thereby eliminating the dependency on same-family architectures. It consistently retains 90%–100% of the original model performance across diverse tasks, with occasional improvements due to denoising effects, while significantly reducing time-to-first-token (TTFT) latency.
📝 Abstract
Prompt length is a major bottleneck in agentic large language model (LLM) workloads, where repeated inference steps and multi-call loops incur substantial prefill cost. Recent work on speculative prefill demonstrates that attention-based token importance estimation can enable training-free prompt compression, but this assumes the existence of a draft model that shares the same tokenizer as the target model. In practice, however, agentic pipelines frequently employ models without any smaller in-family draft model. In this work, we study cross-family speculative prefill, where a lightweight draft model from one model family is used to perform prompt compression for a target model from a different family. Using the same speculative prefill mechanism as prior work, we evaluate a range of cross-family draft-target combinations, including Qwen, LLaMA, and DeepSeek models. Across a broad diversity of tasks, we find that attention-based token importance estimation transfers reliably across different model families despite differences in model architectures and tokenizers between draft and target models. Cross-model prompt compression largely retains 90~100% of full-prompt baseline performance and, in some cases, slightly improves accuracy due to denoising effects, while delivering substantial reductions in time to first token (TTFT). These results suggest that speculative prefill depends mainly on task priors and semantic structure, thus serving as a generalizable prompt compression primitive. We discuss the implications of our findings for agentic systems, where repeated long-context inference and heterogeneous model stacks make cross-model prompt compression both necessary and practical.