CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation

2026-06-23

AI Summary

This work addresses the inefficiencies in serving sparse Mixture-of-Experts (MoE) models, where shared GPU memory between feed-forward network (FFN) weights and key-value (KV) caches leads to low memory utilization, poor long-context support, and resource wastage. To overcome these limitations, we propose a novel dual-memory-pool architecture that decouples model weights from KV caches, integrating KV cache virtualization with a dedicated planner, hierarchical pipeline scheduling, persistent CUDA kernels, and offloaded control flow. This design enables efficient multi-model serving and localized attention computation. Evaluated against state-of-the-art KV cache-based multi-LLM serving systems, our approach reduces P99 time-to-first-token latency by up to 10.4×, substantially enhancing long-context processing capabilities and responsiveness to bursty requests for cold MoE models.

Abstract

Emerging LLM services increasingly host many sparse MoE models, yet most models receive sparse requests and remain cold. This creates a GPU memory problem: model weights are stable and model-determined, while KV-cache is transient and demand-determined. Because cold models rarely reach peak KV-cache demand at the same time, reserving worst-case KV capacity per model wastes memory; a shared KV-cache pool can instead provision aggregate active demand. However, KV-cache sharing is not sufficient when weights and KV-cache remain in a monolithic GPU memory pool. Static weights compete with dynamic KV-cache, and KV-head-limited attention under cold, low-concurrency traffic exposes only a fraction of replicated KV capacity, leading to low GPU memory utilization and weak long-context support. We present CrossPool, a serving engine for cold MoE models that separates FFN weights and KV-cache into two GPU memory pools: a weights pool that consolidates FFN weights across cold models, and a KV-cache pool that dynamically serves active requests while keeping attention local to KV-cache. CrossPool combines a KV-cache planner and virtualizer, a layer-wise pipeline scheduler that hides hidden-state transfers, and persistent kernels with control lowering to reduce CPU-GPU control overhead. With efficient GPU memory pooling, CrossPool underpins bursty long-context requests and outperforms the state-of-the-art kvcached-based multi-LLM serving system, reducing P99 TBT by up to $10.4\times$.

Problem

• cold MoE models
• KV-cache
• GPU memory
• multi-LLM serving
• memory disaggregation

Innovation

• KV-cache disaggregation
• weight pooling
• cold MoE models
• multi-LLM serving
• memory efficiency