General Non-Clairvoyant KV-Cache Scheduling via Regime-Aware Routing

2026-07-10

AI Summary

This work addresses the non-clairvoyant scheduling problem for batched inference of large language models under strict KV cache memory constraints, where prompt lengths are known but response lengths are unknown, with the goal of minimizing total completion time. The authors propose a mechanism-aware routing framework that employs a meta-scheduler to dynamically allocate memory budgets among multiple specialized sub-schedulers. At each decoding step, the framework selects a feasible batch of requests, adaptively accommodating their heterogeneous memory growth patterns. This approach yields the first online scheduling algorithm achieving a constant competitive ratio for arbitrary prompt and response lengths without additional assumptions. It provides theoretical guarantees on both total completion time and makespan, closely approximating the performance of an ideal clairvoyant scheduler under stringent memory limits.

Abstract

We study non-clairvoyant scheduling for batched Large Language Model (LLM) inference under a hard Key-Value (KV) cache memory budget. Each request has a known prompt length but an unknown response length, and its memory footprint comprises a fixed prompt component together with a response component that grows with each decoded token. At each decoding round, the scheduler chooses a feasible batch of active requests; evicting a request discards its accumulated cache states, wasting prior computation. The goal is to minimize total completion time against the optimal clairvoyant schedule that knows all response lengths. We present the first constant-competitive algorithm for arbitrary prompt lengths and arbitrary response lengths with no additional assumptions. Rather than relying on a single universal scheduling policy, our algorithm is built on a novel regime-aware routing framework. Specialized sub-schedulers handle different memory-growth geometries, while a meta-scheduler time-shares the memory budget across them and dynamically routes each job as its execution progressively reveals its behavior. This framework also yields constant-competitive guarantees for makespan and for total completion time under online arrivals.

Problem

• non-clairvoyant scheduling
• KV-cache
• LLM inference
• memory budget
• completion time

Innovation

• non-clairvoyant scheduling
• KV-cache
• regime-aware routing
• constant-competitive algorithm
• LLM inference