vToken: Token-Level Virtualization for Reclaimable KV Caches

2026-08-13

AI Summary

This work addresses the severe memory bottleneck in large language model inference caused by KV cache growth with sequence length and batch size. Existing block-based management schemes, such as PagedAttention, are incompatible with token-level eviction strategies, leading to intra-block fragmentation and inefficient memory reclamation. To overcome this, we propose vToken—the first token-granular KV cache virtualization framework compatible with existing inference engines. vToken employs indirect addressing via a token table to preserve logical cache views while asynchronously compacting active tokens to enable efficient physical memory recovery. It seamlessly supports both PagedAttention and CUDA Graph. Implemented in vLLM, vToken reduces per-request KV blocks by 27.2%–72.3% compared to a Naive-Evict baseline, achieves up to 1.37× higher throughput under SLA constraints, doubles concurrency under a fixed KV memory budget, and simplifies eviction strategy integration from over 500 lines of code to fewer than 50.

Abstract

Large language model serving faces a critical memory bottleneck: the KV cache grows with sequence length and batch size. PagedAttention uses fixed-size memory blocks to reduce allocator-level fragmentation, but recent KV eviction algorithms operate at a token granularity finer than block-level management. This mismatch causes intra-block fragmentation, leaving a large fraction of allocated KV memory unreclaimable. We present vToken, a lightweight token-level virtualization layer that decouples logical token liveness from physical block placement. vToken maintains a stable logical token view through token-table indirection and realizes physical reclamation by repacking live tokens asynchronously. The design preserves PagedAttention kernels and CUDA Graph compatibility. We implement vToken in vLLM and evaluate it with H2O, Random, and Scissorhands across models. Compared with a paired Naive-Evict baseline, vToken reduces retained KV blocks per request by 27.2\%--72.3\% and improves SLA-constrained throughput by up to 1.37$\times$. Under a constrained active-KV budget, it extends the maximum feasible concurrency by up to 2$\times$, while reducing the per-policy integration footprint from 500+ lines to under 50.

Problem

• KV cache
• memory fragmentation
• token-level eviction
• memory reclamation
• large language model serving

Innovation

• token-level virtualization
• KV cache reclamation
• memory fragmentation
• PagedAttention
• asynchronous repacking