Who Should Own the Expert Cache? Kernel-Managed Tiering for Trillion-Parameter MoE Inference

2026-08-12

AI Summary

This work addresses the challenge of managing expert pools in trillion-parameter sparse Mixture-of-Experts (MoE) models, which far exceed GPU memory capacity, rendering existing user-space caching mechanisms inefficient and overly complex. The authors propose a kernel-managed expert caching approach featuring a cooperative design wherein the operating system kernel handles eviction while the model controls admission and provides eviction hints. Leveraging Linux kernel primitives—including the page cache, Multi-Generational LRU (MGLRU), memory ballooning, cgroups, and readahead—the system is evaluated on GH200 nodes using real-world routing traces. Experimental results demonstrate that, under identical memory constraints, the unmodified kernel LRU achieves a 75.3% hit rate—comparable to an ideal frequency-aware policy (74.6%)—yielding a 1.09–1.10× end-to-end inference speedup with bit-identical outputs, thereby providing the first systematic validation of kernel caching as both effective and competitive for MoE inference.

Abstract

Mixture-of-experts models whose expert pools dwarf DRAM force every serving system to contain a cache, yet existing systems typically implement this cache in user space using expert-granular, frequency-ranked, explicitly pinned tiers. We characterize the alternative already provided by the OS: the page cache as the expert tier. We use router traces from three MoE models spanning 128 to 896 experts per layer, including a production trillion-parameter model with a 1.45 TB expert pool, and replay them natively against the full pool on GH200 nodes with capacity enforced by three independent mechanisms. We find four main results. First, iteration time and device traffic are smooth, reproducible functions of cache capacity, making DRAM a practical sizing knob for trillion-parameter serving. Second, the deep-pressure knee is a reclaim artifact requiring both MGLRU and balloon-style mostly-mlocked memory; cgroup limits and physical-memory configurations show no such amplification, indicating that balloon-based studies can overstate pressured-end device traffic by approximately 2$\times$. Third, under an enforced equal-memory wall, untuned kernel LRU serves essentially as much demand as a same-domain oracle frequency table (75.3% versus 74.6% at 256 GB), while the oracle's mechanism advantage is only 1.09$\times$ and disappears off-domain, where LRU remains at 70--71% hit rate. Fourth, router lookahead with 64.7% measured recall provides only a 0.3% benefit as kernel readahead advice and no benefit as synchronous prefetch. End to end, enabling kernel-managed caching improves decode performance by 1.09--1.10$\times$ with token-identical outputs across nine balanced pairs. The resulting design principle is simple: in this regime, let the kernel own eviction, while model-specific knowledge is best spent on admission and advice.

Problem

• Mixture-of-Experts
• expert caching
• kernel-managed tiering
• trillion-parameter models
• inference serving

Innovation

• kernel-managed caching
• Mixture-of-Experts
• page cache
• memory tiering
• trillion-parameter inference