🤖 AI Summary
Current large language model (LLM) inference services rely on generic heuristic strategies that overlook the unique dynamic structure of LLMs in request routing, scheduling, and KV cache management, leading to unstable performance and a lack of theoretical guarantees. This work presents the first systematic integration of operations research and machine learning systems to formally model the distinctive characteristics of LLM inference. Building upon this foundation, we propose an algorithmic framework grounded in mathematical optimization, queueing theory, and cache policy modeling. Our approach matches or surpasses existing heuristics across diverse workloads while providing provable performance bounds and enhanced predictability, thereby establishing a theoretically principled paradigm for LLM serving.
📝 Abstract
This position paper argues that LLM inference serving has outgrown generic heuristics and now demands mathematical optimization and algorithmic foundations. Despite rapid advances in serving systems such as vLLM and SGLang, their algorithmic cores remain largely unchanged from classical distributed computing: request routing uses join-shortest-queue or round-robin, scheduling defaults to FIFO, and KV cache eviction follows LRU. These general-purpose policies ignore the distinctive structure of LLM inference--dynamically growing KV cache memory, prefill-decode phase asymmetry, unknown output lengths, and continuous batching constraints. We contend that the field must develop mathematical models capturing these characteristics, enabling the design of algorithms with provable performance guarantees across diverse workloads, rather than heuristics that may succeed in some scenarios but fail unpredictably in others. Emerging work at the intersection of operations research and ML systems demonstrates that principled methods can match or exceed heuristic performance while providing theoretical guarantees. We call on the community to recognize algorithmic design for LLM serving as a research frontier.