Dynamic HBM Repartitioning for Multi-Turn MoE Serving

2026-09-11

AI Summary

为解决多轮MoE模型服务中GPU内存不足的问题，提出VAMP框架动态调整HBM分区，通过比较三种策略的预估工作量来优化内存使用。

Abstract

Long-running multi-turn requests accumulate reusable key-value (KV) state. Once this state exceeds a fixed GPU KV-cache allocation, serving systems evict reusable prefixes, repeat prefill work, and may preempt requests. This pressure is particularly acute for Mixture-of-Experts (MoE) models: their expert weights occupy most GPU high-bandwidth memory (HBM), even though each token activates only a sparse subset of experts. A static boundary between weights and the KV cache prevents serving systems from using expert memory to preserve reusable state as conversations grow. We present VAMP, an MoE serving framework that changes this HBM boundary at runtime. When a KV allocation cannot be satisfied, VAMP compares the estimated future work of three alternatives: staging expert weights from host memory, evicting cached prefixes that may require re-prefill, or preempting and rescheduling requests. It then converts a bounded expert-weight region into KV-cache capacity when that action has the lowest estimated penalty. CUDA Virtual Memory Management page remapping performs this conversion without copying resident KV data. We implement VAMP in the vLLM serving engine and evaluate Qwen3-Next-80B on recorded and controlled multi-turn workloads. Across five replays of a recorded 2,103-turn SWE-bench agent workload, VAMP with a 15% maximum expert-offloading ratio reduces time-to-first-token (TTFT) p90 from 26.1 s to 1.10 s (23.6 times) and increases request throughput by 20.7% relative to unmodified vLLM, while increasing time per output token (TPOT) by 31.1%.

Problem

• Multi-Turn Requests
• Mixture-of-Experts (MoE) Models
• High-Bandwidth Memory (HBM)
• Key-Value (KV) State
• Prefill Work

Innovation

• Dynamic HBM Repartitioning
• Mixture-of-Experts (MoE)
• CUDA Virtual Memory Management
• Runtime Adjustment