Meta-Soft: Leveraging Composable Meta-Tokens for Context-Preserving KV Cache Compression

2026-05-21

AI Summary

This work addresses the memory and efficiency bottlenecks in large language models caused by the linear growth of KV cache during long-context reasoning. Existing fixed Soft Token compression methods suffer from a lack of input adaptivity and incur irreversible information loss. To overcome these limitations, the authors propose Meta-Soft, a novel framework that introduces a composable meta-token mechanism. It constructs a learnable orthogonal basis as a meta-bank and employs a Gumbel-Softmax selector to dynamically synthesize k input-aware Soft Tokens for probing salient information. Coupled with an attention-flow-driven semantic redistribution strategy, the framework reallocates information from pruned tokens to retained ones. This approach enables context-aware KV cache compression, achieving state-of-the-art performance across multiple benchmarks while preserving both computational efficiency and semantic fidelity.

Abstract

The KV cache used in large language models has linearly growing time complexity, so LLMs face memory blow-up and reduced decoding efficiency when they process long contexts.Current KV Cache eviction has become an important research direction; however, existing methods based on fixed Soft Tokens (e.g., Judge Q) rely on a static parameter set as the query to evaluate the importance of KV pairs, so they cannot adapt dynamically to different input prompts, and they cannot precisely capture complex and changing task relevance.Also, evicted KV pairs are discarded permanently, so this causes irreversible information loss and context breaks. To address this problem, we propose Meta-Soft, a dynamic compression framework based on probe-driven context integration. Specifically, we build a meta-library with a learnable orthogonal basis matrix $\mathcal{L}$, and we use a selector network with Gumbel-Softmax to produce differentiable sparse combination weights, so we dynamically synthesize the most targeted $k$ Soft Tokens from the input prompt features.We append these Soft Tokens to the end of the input sequence to probe key information. We also introduce an attention-flow based integration mechanism, which redistributes the semantic information of removed tokens into retained tokens, and this keeps the dropped context information effectively.Experiments on multiple datasets show that our method outperforms existing state-of-the-art eviction methods and provides a new solution for KV Cache compression.

Problem

• KV cache compression
• long-context processing
• memory efficiency
• context preservation
• dynamic token eviction

Innovation

• KV cache compression
• dynamic Soft Tokens
• context preservation
• attention flow
• meta-learning