KVEraser: Learning to Steer KV Cache for Efficient Localized Context Erasing

2026-06-15

AI Summary

This work addresses the challenge of efficient local KV cache editing in long-context large language models, where naive deletion of cached key-value pairs induces global dependencies, necessitating prohibitively expensive full recomputation of subsequent tokens. To overcome this, the authors propose a learnable KV cache editing method that replaces the deleted region with a trainable proxy state while preserving the rest of the cache intact. The approach employs a two-stage training strategy: neighborhood-aware pretraining to mitigate deletion artifacts and task-specific fine-tuning to adapt to downstream applications, enabling transferable and efficient edits. Evaluated across context lengths from 1K to 32K, the method achieves performance nearly matching full recomputation with only a 24% latency overhead—dramatically lower than the 17.6× cost of exact recomputation—and demonstrates a 3–4× speedup with state-of-the-art accuracy on unseen long-document question answering tasks.

Abstract

Post-hoc context erasing over the KV cache is challenging because a local edit has a global consequence: once a span has been processed, its influence propagates into the cached states of all subsequent tokens. This issue arises naturally in long-context LLM applications, where stale retrieved facts, incorrect tool observations, retracted user preferences, or harmful prompt injections may be identified only after prefill. Exact erasing must then recompute all tokens after the deleted span, making its computational cost depend on suffix length rather than erased-span length. We introduce KVEraser, a learned KV-cache editing method for efficient localized context erasing. Given a processed context and a span to remove, KVEraser replaces only the KV states of the erased interval with learned steering states while reusing the remaining cache unchanged. To learn a transferable erasing mechanism, we build a two-stage training pipeline: generic span-neighbor pre-training teaches the eraser to suppress the influence of the erased span, while task-specific fine-tuning adapts this capability to downstream scenarios. Experiments show that KVEraser nearly matches full recomputation in post-erasure performance on in-domain tasks across 1K--32K context lengths, while its latency increases by only 24% compared with a 17.6x increase for full recomputation. KVEraser also generalizes to unseen long-document QA tasks with harmful factual distractors, achieving the best performance among approximate baselines with a 3--4x speedup over full recomputation.

Problem

• KV cache
• context erasing
• long-context LLM
• post-hoc editing
• efficient inference

Innovation

• KV cache editing
• localized context erasing
• post-hoc editing
• efficient inference
• transferable erasing