Beyond Prediction: Tail-Aware Scheduling for LLM Inference

2026-06-16

AI Summary

In large model inference serving, the high variability of request lengths undermines the robustness of prediction-based scheduling under distribution shifts, bursty workloads, and GPU memory pressure, while also making tail latency (P90–P99) difficult to optimize. This work proposes a length-prediction-free scheduling framework that leverages lightweight statistical signals to enable soft priority promotion, co-designed with a cache-aware preemption mechanism and a memory-coupled decoding dynamics model. Evaluated on both real-world and open-source traces, the proposed approach reduces P99 time-to-first-token (TTFT) by 34–47% and total time-to-last-token (TTLT) by 35–50% compared to SRPT equipped with perfect length information, substantially improving tail latency performance and system robustness.

Abstract

LLM serving exhibits extreme length variability, making size-based scheduling difficult in practice. Recent LLM schedulers approximate SJF/SRPT using predicted decode lengths or ranks and primarily report mean-centric metrics such as TTFT and TBT. We show that these prediction-driven policies can be fragile under distribution shifts, bursty arrivals, and GPU memory pressure, while offering limited control over the tail latency (P90-P99) that dominates user experience, even with perfect decode-length knowledge. We introduce a distribution-aware, prediction-free scheduling framework that replaces explicit length prediction with soft priority boosting driven by lightweight statistical signals. Our design co-optimizes scheduling and cache-aware preemption to account for memory-coupled decode dynamics across workload mixes. Evaluated on production and open-source traces, our method reduces P99 TTLT by up to 35-50% relative to SRPT with perfect length knowledge and reduces TTFT by 34-47% across workloads, including reasoning-heavy and chat-heavy tasks. These results demonstrate a robust alternative for optimizing tail latency in online LLM serving.

Problem

• LLM inference
• tail latency
• scheduling
• distribution shift
• GPU memory pressure

Innovation

• tail-aware scheduling
• prediction-free
• LLM inference
• cache-aware preemption
• distribution-aware