Tropical: Enhancing SLO Attainment in Disaggregated LLM Serving via SLO-Aware Multiplexing

2026-06-15

AI Summary

This work addresses the challenge that conventional decoupled and non-decoupled architectures in large language model (LLM) serving struggle to simultaneously meet service-level objectives (SLOs) for time-to-first-token (TTFT) and time-per-output-token (TPOT). To overcome this limitation, the authors propose an SLO-aware dynamic multiplexing scheduling mechanism that builds upon the decoupled architecture to jointly optimize queuing delays and execution interference across the prefill and decode stages. This approach is the first to integrate SLO-driven multiplexing into LLM serving scheduling, effectively combining the strengths of both architectural paradigms. Experimental results on real-world datasets demonstrate that the method improves the fraction of SLO-compliant requests by up to 2.09× at the 90th percentile; further, it achieves a 9× reduction in P90 TTFT with only a 15% TPOT trade-off, or alternatively enhances P90 TPOT by 2.8× while keeping TTFT unchanged.

Abstract

To guarantee service quality in transformer based large language model (LLM) serving, it is essential to meet the latency constraints of both the prefill phase (measured by Time-to-First-Token, TTFT) and the decode phase (measured by Time-per-Output-Token, TPOT). Non-disaggregated serving places prefill and decode on the same worker, while disaggregated serving places the prefill and decode on isolated workers. However, no single architecture excels in both TTFT and TPOT metrics. After conducting a root cause analysis, we concluded that indisaggregated LLM serving, prefill execution has minimal interference with decode execution but result in high queuing times. In contrast,non-disaggregated LLM serving effectively reduces queuing times but introduces significant interference between prefills and decodes. In order to leverage the best aspects of both non-disaggregated anddisaggregated LLM serving, we have designed and implemented Tropical.Tropical introduces an sevice-level objectives (SLO)-aware multiplexing strategy that balances the queuing time and the interference, enabling the LLM serving to achieve high TTFT and TPOT SLOs simultaneously. Our evaluation of real-world datasets reveals that Tropical outperforms both state-of-the-art non-disaggregated and disaggregated LLM serving systems, achieving up to 2.09 more requests within a 90% SLO attainment. Specially, compared to the disaggregated LLM serving system, Tropicalimproves P90 TTFT performance by 9 with only an 15% reduction in P90 TPOT. Against the non-disaggregated LLM serving systems, Tropicaldelivers a 2.8 performance improvement in P90 TPOT while maintaining the same P90 TTFT.

Problem

• LLM serving
• SLO attainment
• disaggregated architecture
• latency constraints
• TTFT/TPOT

Innovation

• SLO-aware multiplexing
• disaggregated LLM serving
• latency optimization
• TTFT/TPOT balancing
• interference mitigation