NestedKV: Nested Memory Routing for Long-Context KV Cache Compression

2026-05-26

AI Summary

This work addresses the high memory overhead of KV caching in long-context language models, where existing training-free compression methods—relying on a single importance signal—struggle with globally unique, locally fragmented, or immediately relevant context. The authors propose NestedKV, a training-free, key-only compression method that introduces a novel nested memory routing mechanism. By leveraging three tiers of key anchors—global, block-level, and sliding window—combined with multi-timescale cosine anomaly scoring, head-adaptive budget allocation, and surprise-gated token routing, NestedKV significantly enhances performance under tight cache budgets without modifying the model or requiring additional training. On Qwen3-4B, it outperforms KeyDiff by 19.10 and 19.29 points on RULER and LongBench at compression ratio r=0.75, and achieves a LongBench score of 37.32 versus KeyDiff’s 17.55 at r=0.95, setting new state-of-the-art results across multiple long-context benchmarks.

Abstract

Long-context language models are limited by the memory footprint of the key-value (KV) cache. Existing training-free KV compression methods usually rank tokens by one importance signal -- attention, recency, layer-wise allocation, or key distinctiveness -- which becomes brittle when useful context is globally distinctive, locally episodic, or immediately relevant. We introduce NestedKV, a key-only KV cache compression method inspired by the Continuum Memory System in Nested Learning. NestedKV maintains global, block-level, and sliding-window key anchors, scores tokens by multi-time-scale cosine anomaly, and combines the resulting rankings with a training-free outer learner using head-adaptive mixing and surprise-gated token routing. The score is paired with adaptive per-head budgets and requires no training or LLM modification. Across RULER (4k--32k), LooGLE, LongBench, LongBench-E, InfiniteBench, and MMLU-Pro on Qwen3 and Llama-3.2 models, NestedKV is strongest when the retained cache is small. On Qwen3-4B, it improves over KeyDiff by up to 19.10 points on RULER and 19.29 on LongBench at $r=0.75$; at $r=0.95$, it retains 37.32 on LongBench versus 17.55 for KeyDiff.

Problem

• KV cache compression
• long-context language models
• memory footprint
• token importance
• key-value cache

Innovation

• KV cache compression
• nested memory routing
• multi-time-scale scoring
• training-free compression
• long-context LLMs