Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice

2026-06-21

AI Summary

This work addresses the inefficiency of conventional time-based scheduling strategies in large language model (LLM) inference, which fail to capture the dynamic memory footprint and two-dimensional spatiotemporal geometric growth of Key-Value (KV) caches. The authors formulate LLM inference scheduling as a geometric volume minimization problem and propose Smallest Volume First (SVF), a geometry-aware online scheduling algorithm, along with 1-bit SVF—a lightweight variant requiring only one bit of metadata per request. Theoretical analysis tightens the worst-case competitive ratio under known output lengths from 48 to 5 and introduces a taxonomy of information availability across scheduling scenarios. Integrated as a plug-in into vLLM, SVF significantly reduces both average and tail latency on Llama-3.1, while 1-bit SVF achieves throughput and latency performance comparable to state-of-the-art schedulers with minimal overhead.

Abstract

The explosive demand for interactive Large Language Model serving has highlighted the management of the Key-Value cache's dynamic memory footprint as a critical area for performance optimization in inference engines. Modern inference systems overwhelmingly rely on time-centric scheduling heuristics, such as Shortest Job First. However, their theoretical optimality is rooted in traditional schedule modeling, failing to capture the highly dynamic, 2D spatio-temporal geometric growth specific to LLM inference mechanisms. To resolve this, we propose the geometry-aware online scheduling by introducing the Smallest Volume First (SVF) algorithm and its highly efficient variant, 1-bit SVF. Theoretically, we provide a rigorous mathematical foundation for our approach. Utilizing a novel proof methodology, we tighten the worst-case competitive ratio ($\text{CR} \le 48 \rightarrow \text{CR} \le 5$) for SVF with known output lengths. Building upon this core breakthrough, we complete a comprehensive theoretical taxonomy analyzing our algorithms across different traffic scenarios and information availability. Practically, we seamlessly integrate our approach as a plug-and-play layer in vLLM. Extensive evaluations on Llama-3.1 models demonstrate comprehensive performance gains: SVF delivers strong reductions in both average and tail latency, while 1-bit SVF, with merely a single bit information, achieves competitive throughput and latency. This work establishes a theoretically sound and empirically proven approach for resolving memory-constrained scheduling in modern LLM deployments. To facilitate future research, our code is available at https://github.com/Aurora-Kl/Geometry-Aware-Online-Scheduling.git.

Problem

• LLM serving
• Key-Value cache
• online scheduling
• memory-constrained scheduling
• geometry-aware

Innovation

• geometry-aware scheduling
• Key-Value cache
• Smallest Volume First
• competitive ratio
• LLM serving