PRISM: Fast Online LLM Serving via Scheduling-Memory Co-design

2026-05-08

AI Summary

This work addresses the low key-value (KV) cache reuse efficiency in online large language model serving, caused by prompt fragmentation and repetitive hot prefixes, which leads to redundant prefill computations and increased time-to-first-token (TTFT). To overcome the disconnect between request scheduling and KV cache management in existing approaches, the authors propose a co-optimized framework that integrates a query-aware scheduler (QAS) with a demand-aware radix tree (DART), enabling aligned request scheduling and precise prefix-based KV caching. Experimental results demonstrate that, compared to the strongest baseline, the proposed method reduces P99 TTFT by 23.3% and 37.1% on 4B and 13B parameter models, respectively, while improving exact-prefix cache hit rates by 5.9 and 12.2 percentage points.

Abstract

Modern online large language model (LLM) services, such as Retrieval-Augmented Generation (RAG) and agent systems, increasingly expose two prominent characteristics: prompt segmentation (e.g., system instructions, retrieved passages, tool outputs) and hotspot skew, where a small set of these segments recurs frequently across user requests. Failing to jointly exploit these patterns could lead to repeated prefill of hot segments and prolonged TTFT, undermining both throughput and user-perceived responsiveness. However, existing work tackles these patterns independently: KV-cache management mainly exploits segment reuse while scheduling reorders requests to improve cache locality, yet neither aligns request admission with KV-cache retention. To address this gap, we first analyze how scheduling and KV-cache management jointly affect TTFT. Guided by this, we present PRISM (Prefix Reuse Optimization Integrated Scheduling and Memory), which co-designs a query-aware scheduler (QAS) with a demand-aware radix tree (DART) to align request admission with exact-prefix KV retention. Our evaluation results show that, versus the strongest baseline, PRISM reduces average per-QPS P99 TTFT by 23.3\% and 37.1\% while increasing exact-prefix KV-cache hit rate by 5.9 and 12.2 percentage points on 4B and 13B models, respectively.

Problem

• prompt segmentation
• hotspot skew
• KV-cache reuse
• TTFT
• online LLM serving

Innovation

• scheduling-memory co-design
• KV-cache reuse
• prompt segmentation
• hotspot skew
• TTFT optimization