Adaptive Context Parallelism for Production LLM Serving

2026-09-04

AI Summary

为解决长序列输入导致的计算和内存需求增加问题，提出了一种自适应上下文并行服务系统Vertumnus，通过动态调整并行度和全局缓存管理策略来优化性能。

Abstract

As LLM context windows expand and input sequences grow longer, serving systems face increasing computational and memory demands. Context parallelism (CP), which partitions the input sequence across multiple ranks to parallelize the computation, has therefore become increasingly important for efficient LLM serving. However, existing CP-enabled systems either rely on static CP configurations or adjust the CP degree only for active requests or batches. In this paper, we present Vertumnus, an adaptive CP serving system designed for heterogeneous and evolving workloads. At the request level, Vertumnus routes requests among workers with different CP degrees using a placement cost that combines predicted queuing delay, cache-aware prefill time, and GPU-time cost. At the cluster level, Vertumnus adapts the worker composition through seconds-scale split and merge operations as workload demand changes. Vertumnus further introduces a global prefix-cache management policy that coordinates cache placement and replication among workers with the same or different CP degrees, preserving cache locality as request assignments and worker composition change. Experiments on a 64-GPU cluster with public and production workloads show that, under the highest evaluated loads, Vertumnus reduces mean TTFT by up to 28.1% and improves token-weighted SLO attainment by up to 13.3 percentage points over the strongest baseline.

Problem

• Adaptive Context Parallelism
• Production LLM Serving
• Heterogeneous Workloads

Innovation

• Adaptive Context Parallelism
• Heterogeneous Workloads
• Global Prefix-Cache Management