Online Linear Programming for Multi-Objective Routing in LLM Serving

2026-07-04

AI Summary

This work addresses the challenge of real-time request routing in large language model serving, where incoming requests must be scheduled to decoding nodes under constraints on batch size and KV cache capacity. Existing heuristic approaches struggle to explicitly balance the trade-off between latency and throughput. To overcome this limitation, the paper introduces a novel framework that integrates multi-objective optimization with online linear programming. Request admission decisions are made by comparing SLO-weighted rewards against dual shadow prices, while a warm-started first-order projected gradient method efficiently tracks dynamic dual variables, enabling millisecond-scale, interpretable, and tunable scheduling. Experiments on the Vidur simulation platform demonstrate that the proposed approach consistently outperforms baseline methods across diverse SLO configurations, achieving comprehensive improvements in end-to-end latency, time-to-first-token, throughput, and tail performance.

Abstract

We study the online routing problem in large language model serving, where requests arrive sequentially and must be dispatched to parallel decode workers under tight batch-size and KV-cache constraints. Unlike widely used routing heuristics that are not tied to explicit service-level objectives (SLOs) and offer limited control over latency-throughput trade-offs, we introduce a multi-objective optimization framework that formulates routing as an online linear programming with interpretable decision rewards. We apply an efficient bid-price control policy based on the online linear programming that admits requests when their SLO-weighted benefit exceeds their shadow prices. To meet millisecond decision requirements, we develop a warm-started, projected first-order updates that track the evolving dual shadow prices online with predictable runtime. We integrate our router into the Vidur simulator and demonstrate substantial improvements over standard baselines across multiple SLO regimes, including end-to-end latency, time-to-first-token, throughput, and tail performance. A big picture from our result: a science-based approach outperforms others based on heuristics.

Problem

• online routing
• multi-objective optimization
• large language model serving
• service-level objectives
• online linear programming

Innovation

• online linear programming
• multi-objective routing
• SLO-aware serving
• bid-price control
• first-order optimization