SCORPIO: Serving the Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference

2025-05-29

AI Summary

Existing LLM serving systems prioritize throughput over SLO compliance, resulting in low达标 rates for TTFT (Time-To-First-Token) and TPOT (Time-Per-Output-Token). This paper proposes an SLO-oriented heterogeneous scheduling framework featuring a novel dual-guard mechanism: the TTFT Guard enforces deadline-aware request reordering and rejects infeasible requests, while the TPOT Guard integrates Virtual Batch Size (VBS)-based admission control with credit-driven dynamic batching. A lightweight, SLO-aware online prediction module orchestrates joint optimization of goodput and SLO attainment across admission, queuing, and batching stages. Experiments demonstrate that, compared to state-of-the-art baselines, our system achieves up to a 14.4× improvement in goodput and up to a 46.5% increase in the joint TTFT/TPOT SLO compliance rate.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Hardware-aware ML
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Existing Large Language Model (LLM) serving systems prioritize maximum throughput. They often neglect Service Level Objectives (SLOs) such as Time to First Token (TTFT) and Time Per Output Token (TPOT), which leads to suboptimal SLO attainment. This paper introduces SCORPIO, an SLO-oriented LLM serving system designed to maximize system goodput and SLO attainment for workloads with heterogeneous SLOs. Our core insight is to exploit SLO heterogeneity for adaptive scheduling across admission control, queue management, and batch selection. SCORPIO features a TTFT Guard, which employs least-deadline-first reordering and rejects unattainable requests, and a TPOT Guard, which utilizes a VBS-based admission control and a novel credit-based batching mechanism. Both guards are supported by a predictive module. Evaluations demonstrate that SCORPIO improves system goodput by up to 14.4X and SLO adherence by up to 46.5% compared to state-of-the-art baselines.

Problem

• Optimizing LLM serving for heterogeneous SLOs
• Improving Time to First Token (TTFT) adherence
• Enhancing Time Per Output Token (TPOT) efficiency

Innovation

• Adaptive scheduling for heterogeneous SLOs
• TTFT Guard with deadline-first reordering
• TPOT Guard with credit-based batching