Multi-stage Flow Scheduling for LLM Serving

2026-03-18

AI Summary

This work addresses severe network contention in large language model (LLM) serving caused by overlapping traffic from multi-stage workflows, which degrades time-to-first-token (TTFT) latency and compromises service-level objective (SLO) compliance. To mitigate this, the authors propose MFS, a novel scheduling mechanism that introduces stage-aware scheduling to LLM inference for the first time. Built upon a reverse multi-level queue (RMLQ) architecture, MFS employs a Defer-and-Promote strategy to approximate Least-Laxity-First scheduling without requiring precise knowledge of remaining slack time. This dynamically elevates the priority of urgent requests while preventing those with loose SLOs from prematurely consuming bandwidth. Implemented as a plug-in module in vLLM, MFS improves TTFT SLO attainment by 1.2–2.4× across an 8-server, 32-GPU testbed and large-scale simulations.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Multiagent Learning

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Meeting stringent Time-To-First-Token (TTFT) requirements is crucial for LLM applications. To improve efficiency, modern LLM serving systems adopt disaggregated architectures with diverse parallelisms, introducing complex multi-stage workflows involving reusable KV-block retrieval, collective communication, and P2D transfer. Flows from dependent stages overlap within and across requests on shared bottleneck links, making TTFT highly susceptible to network contention and necessitating stage-aware scheduling. Unfortunately, most existing works schedule flows in a stage-agnostic manner, leading to uncoordinated contention that constitutes a primary cause of SLO violations. In this paper, we present MFS, a holistic multi-stage flow scheduling mechanism designed to maximize TTFT SLO attainment. At its core, MFS approximates the Least-Laxity-First (LLF) scheduling policy without requiring precise knowledge of a request's remaining slack. It achieves this through a Defer-and-Promote principle implemented through a Reverse Multi-Level Queue (RMLQ) structure. By dynamically promoting task precedence as effective laxity diminishes, MFS prioritizes flows with less laxity while preventing requests with loose SLOs from prematurely consuming network bandwidth. We implement MFS as a pluggable module integrated into vLLM, and evaluate it on a 8-server, 32-GPU testbed as well as through large-scale simulations. Our results demonstrate that MFS effectively outperforms state-of-the-art baselines, improving the TTFT SLO attainment by 1.2x--2.4x.

Problem

• LLM serving
• Time-To-First-Token
• multi-stage scheduling
• network contention
• SLO violation

Innovation

• Multi-stage Flow Scheduling
• Time-To-First-Token
• Least-Laxity-First
• Reverse Multi-Level Queue
• LLM Serving