WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware

2026-06-20

AI Summary

This work addresses the challenge of deploying Mixture-of-Experts (MoE) models on extremely memory-constrained hardware, where inactive experts still consume substantial GPU memory, and conventional CPU offloading negates sparsity benefits due to frequent PCIe transfers. The authors formulate this as a GPU working set management problem and propose WiSP, a routing-aware expert paging mechanism, along with a marginal-value-based working set allocation strategy (MV-WSA), which jointly optimizes expert scheduling and KV cache memory allocation for the first time. Their approach supports both offline configuration and online dynamic adjustment while ensuring byte-level output consistency without modifying the inference engine. Experiments show that when the model cannot fully reside in GPU memory, WiSP achieves up to 1.95× higher decoding throughput than static offloading; MV-WSA balances prefill and decoding performance in real-world serving, with offline configurations approaching ideal allocation—outperforming fixed strategies by ~20%—and an online controller further improving performance by 1.20×.

Abstract

Modern Mixture-of-Experts (MoE) models place most of their parameters in expert layers, yet only a small fraction of those experts are used for any token. The unused weights must still be stored where the GPU can reach them. On commodity GPUs the common fix is layer-level CPU offloading, which keeps memory low but streams all of a layer's experts across PCIe on every forward pass, losing much of MoE's sparsity benefit. We cast low-resource MoE serving as a working-set management problem on the GPU: routed expert weights and the key-value (KV) cache are two streams of memory demand competing for limited VRAM. We realize this in WiSP (Working-Set Paging), a routing-aware expert pager that plugs into an unmodified serving engine with byte-identical outputs. Keeping resident only the experts a workload reuses, WiSP reaches up to 1.95x the decode throughput of static offload at the same memory budget when the model does not fit. We also find that prefetching experts from predicted routing helps little in single-stream decode: the bottleneck is PCIe bandwidth, not prediction accuracy. This shifts the question from prefetching to allocation: how should VRAM be split between experts and the KV cache? We answer with MV-WSA (Marginal-Value Working-Set Allocation), which equalizes marginal latency benefit per byte subject to a KV admission floor. MV-WSA runs either as an offline configurator or as an online controller that resizes both pools while serving. In real serving the offline configurator is the only policy we test that does well on both prefill and decode; in trace-driven simulation it stays within a few percent of a per-workflow oracle while fixed splits are about 20% worse. The online controller adds a further 1.20x without changing model outputs.

Problem

• Mixture-of-Experts
• low-resource serving
• working-set management
• GPU memory allocation
• KV cache

Innovation

• Mixture-of-Experts
• working-set paging
• low-resource serving
• VRAM allocation
• KV cache management