DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing

2025-11-06

AI Summary

In LLM inference services, severe GPU resource interference arises when prefill and decoding co-execute on shared GPUs, degrading tokens-between-time (TBT) latency and hindering simultaneous achievement of high throughput and low latency SLOs. This paper proposes an adaptive Streaming Multiprocessor (SM)-level GPU multiplexing framework. It introduces an attention-aware roofline model for precise per-iteration latency prediction, a fine-grained, demand-driven streaming SM space partitioning mechanism, and a CPU-GPU synchronization-free, interruption-transparent execution engine to dynamically decouple prefill and decoding within a single GPU. Evaluated against state-of-the-art systems under stringent latency SLOs, our approach achieves up to 1.3× higher throughput while maintaining stable generation latency, significantly reducing redundant KV cache transfers and scheduling overhead.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Modern LLM serving systems must sustain high throughput while meeting strict latency SLOs across two distinct inference phases: compute-intensive prefill and memory-bound decode phases. Existing approaches either (1) aggregate both phases on shared GPUs, leading to interference between prefill and decode phases, which degrades time-between-tokens (TBT); or (2) disaggregate the two phases across GPUs, improving latency but wasting resources through duplicated models and KV cache transfers. We present DuetServe, a unified LLM serving framework that achieves disaggregation-level isolation within a single GPU. DuetServe operates in aggregated mode by default and dynamically activates SM-level GPU spatial multiplexing when TBT degradation is predicted. Its key idea is to decouple prefill and decode execution only when needed through fine-grained, adaptive SM partitioning that provides phase isolation only when contention threatens latency service level objectives (SLOs). DuetServe integrates (1) an attention-aware roofline model to forecast iteration latency, (2) a partitioning optimizer that selects the optimal SM split to maximize throughput under TBT constraints, and (3) an interruption-free execution engine that eliminates CPU-GPU synchronization overhead. Evaluations show that DuetServe improves total throughput by up to 1.3x while maintaining low generation latency compared to state-of-the-art frameworks.

Problem

• Balancing prefill and decode phases in LLM serving
• Eliminating GPU interference between inference phases
• Optimizing throughput while maintaining latency SLOs

Innovation

• Adaptive GPU multiplexing for prefill and decode phases
• SM-level spatial partitioning to isolate conflicting phases
• Attention-aware roofline model for latency prediction optimization