Locality-aware Fair Scheduling in LLM Serving

2025-01-24

AI Summary

Addressing the challenge of jointly ensuring fairness and prefix locality in large language model (LLM) serving, this paper proposes the first scheduling framework that simultaneously guarantees strict positional fairness and high KV cache locality. Methodologically, it introduces two algorithms: DLPM for single-node deployment and D²LPM for distributed settings—integrating deficit counting, longest prefix matching (LPM), virtual-time scheduling, and consistent hashing to support dynamic client weights and real-time cache awareness. The core contribution is a three-dimensional co-optimization of fairness (Jain’s fairness index >0.95), prefix locality (measured by improved KV cache hit rate), and load balancing. Experimental results demonstrate that, compared to virtual token counters (VTC), the framework achieves a 2.87× throughput improvement; against state-of-the-art distributed LLM serving systems, it reduces client tail latency by 7.18×.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language model (LLM) inference workload dominates a wide variety of modern AI applications, ranging from multi-turn conversation to document analysis. Balancing fairness and efficiency is critical for managing diverse client workloads with varying prefix patterns. Unfortunately, existing fair scheduling algorithms for LLM serving, such as Virtual Token Counter (VTC), fail to take prefix locality into consideration and thus suffer from poor performance. On the other hand, locality-aware scheduling algorithms in existing LLM serving frameworks tend to maximize the prefix cache hit rate without considering fair sharing among clients. This paper introduces the first locality-aware fair scheduling algorithm, Deficit Longest Prefix Match (DLPM), which can maintain a high degree of prefix locality with a fairness guarantee. We also introduce a novel algorithm, Double Deficit LPM (D$^2$LPM), extending DLPM for the distributed setup that can find a balance point among fairness, locality, and load-balancing. Our extensive evaluation demonstrates the superior performance of DLPM and D$^2$LPM in ensuring fairness while maintaining high throughput (up to 2.87$ imes$ higher than VTC) and low per-client (up to 7.18$ imes$ lower than state-of-the-art distributed LLM serving system) latency.

Problem

• Large-scale Language Model
• Fair Task Scheduling
• Resource Allocation Efficiency

Innovation

• Deficit Longest Prefix Matching
• Distributed Task Scheduling
• Large Language Model Services