Not All Prefills Are Equal: PPD Disaggregation for Multi-turn LLM Serving

2026-03-09

AI Summary

In multi-turn dialogue systems, the conventional Prefill-Decode architecture suffers from high latency and bandwidth bottlenecks due to repeated transmission of KV caches and full prefilling at every turn. This work proposes PPD, a decoupled architecture that explicitly distinguishes between append-prefill and full-prefill operations. It introduces a configurable, SLO-aware dynamic routing mechanism that localizes prefill requests containing only new tokens to decode nodes, thereby reusing existing KV caches. By overcoming the limitations of fixed routing strategies, PPD reduces the time-to-first-token for second and subsequent turns by 68% under high load while maintaining efficient per-output-token latency, significantly alleviating KV cache transmission overhead.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Prefill-Decode (PD) disaggregation has become the standard architecture for modern LLM inference engines, which alleviates the interference of two distinctive workloads. With the growing demand for multi-turn interactions in chatbots and agentic systems, we re-examined PD in this case and found two fundamental inefficiencies: (1) every turn requires prefilling the new prompt and response from the last turn, and (2) repeated KV transfers between prefill and decode nodes saturate the bandwidth, leading to high latency and even service degradation. Our key insight is that not all prefill operations are equally disruptive: append-prefill -- processing only the new input tokens while reusing cached KV states -- incurs substantially less decoding slowdown than full prefill. This motivates routing append-prefill to decode nodes locally. However, through comprehensive analysis, we show that no single fixed routing strategy satisfies all Service Level Objectives (SLOs) simultaneously. Based on this insight, we propose Prefill Prefill-capable Decode (PPD) disaggregation, a dynamic routing system that decides when to process Turn 2+ requests locally on decode nodes using cached KV states. PPD adapts to varying SLOs via configurable weights and seamlessly integrates with traditional PD deployments. With extensive evaluations, we show that PPD reduces Turn 2+ time-to-first-token (TTFT) by 68% while maintaining competitive time-per-output-token (TPOT), effectively alleviating KV transfer congestion under high load. We believe PPD represents a flexible and efficient paradigm for multi-turn LLM serving.

Problem

• multi-turn LLM serving
• Prefill-Decode disaggregation
• KV cache transfer
• latency
• bandwidth saturation

Innovation

• PPD disaggregation
• multi-turn LLM serving
• append-prefill
• KV cache reuse
• dynamic routing