PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Serving

2026-04-13

AI Summary

This work addresses the limitations of existing large language model inference systems that rely on static pipeline parallelism and struggle to adapt to dynamic workloads, as conventional reconfiguration approaches require service-disrupting downtime. The authors propose a novel pipeline parallelism mechanism supporting online, in-place reconfiguration through co-design of KV cache layout and attention computation. By introducing incremental KV state synchronization and an algorithm for identifying safe switching points—built upon an extended PageAttention framework for unified cache management—the system enables dynamic adjustment of parallel configurations without service interruption while preserving state consistency. Experimental results demonstrate substantial performance gains: compared to a baseline disabling KV adjustment, first-token latency is reduced by 2.5×; against a variant without the proposed patching mechanism, reconfiguration latency drops from seconds to under 10 ms, with up to 54.7% and 14.7% improvements in first-token and per-token generation latency, respectively.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Pipeline parallelism (PP) is widely used to partition layers of large language models (LLMs) across GPUs, enabling scalable inference for large models. However, existing systems rely on static PP configurations that fail to adapt to dynamic settings, such as serverless platforms and heterogeneous GPU environments. Reconfiguring PP by stopping and redeploying service incurs prohibitive downtime, so reconfiguration must instead proceed live and in place, without interrupting inference. However, live in-place PP reconfiguration is fundamentally challenging. GPUs are already saturated with model weights and KV cache, leaving little room for new layer placements and necessitating KV cache resizing, at odds with systems like vLLM that preallocate for throughput. Moreover, maintaining KV consistency during execution is difficult: stop-and-copy introduces large pauses, while background synchronization risks inconsistency as states evolve. We present PipeLive, which enables live in-place PP reconfiguration with minimal disruption. PipeLive introduces a redesigned KV cache layout together with a co-designed extension to PageAttention, forming a unified mechanism for live KV resizing. It further adopts an incremental KV patching mechanism, inspired by live virtual machine migration, to synchronize KV states between source and target configurations and identify a safe switch point. PipeLive achieves a 2.5X reduction in time-to-first-token (TTFT) without KV cache overflow compared to disabling KV resizing. Furthermore, compared to a variant without KV patching, it reduces reconfiguration overhead from seconds to under 10ms, and improves TTFT and time-per-output-token (TPOT) by up to 54.7% and 14.7%, respectively.

Problem

• pipeline parallelism
• live reconfiguration
• KV cache
• dynamic LLM serving
• in-place

Innovation

• live reconfiguration
• pipeline parallelism
• KV cache resizing
• incremental KV patching
• PageAttention