Forget Without Compromise: Nexus Sampling for Streaming KV-Cache Eviction Under Fixed Budgets

2026-06-22

AI Summary

This work addresses the challenge of KV cache overflow in long-context and agent-style large language model inference, where existing token eviction methods often mistakenly discard low-attention yet critical bridging tokens. To overcome this limitation, the authors propose Nexus Sampling—a training-free KV cache eviction strategy that uniquely integrates iterative attention path analysis with weighted reservoir sampling. By assigning a Nexus score to each token and retaining them probabilistically based on this metric, the method transcends the rigidity of conventional deterministic top-K eviction. Experimental results demonstrate that even at an 80% cache eviction rate, Nexus Sampling incurs only a 1% performance drop relative to dense attention on LongBench, significantly outperforms baseline approaches on retrieval-intensive tasks, and reduces per-sequence memory consumption by up to 10×.

Abstract

Long-context and agentic LLM workloads push the KV cache past any fixed memory budget, forcing the inference stack to permanently evict tokens at every step of a continuous-inference stream. Existing methods all share the same template, a per-step direct-attention score followed by deterministic top-$K$ selection, which converts a single below-cutoff step into an irreversible verdict and permanently erases any subtly important token that direct attention cannot single out from noise. To address this challenge, we propose Nexus Sampling, a training-free eviction method that pairs Nexus scoring, an iterative walk over direct attention that surfaces bridge tokens, with weighted reservoir sampling, which retains tokens with inclusion probability in place of deterministic top-$K$. Theoretically, we show that Nexus Sampling dominates deterministic top-$K$ in long-run survival of subtly important tokens. Empirically, at 80% KV cache eviction, Nexus Sampling matches dense attention within 1% on LongBench while outperforming top-$K$ baselines on retrieval-heavy tasks, with up to 10x smaller per-sequence cache memory.

Problem

• KV-cache eviction
• streaming inference
• fixed memory budget
• long-context LLMs
• token importance

Innovation

• Nexus Sampling
• KV-cache eviction
• weighted reservoir sampling
• long-context LLM
• streaming inference