AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reuse

2026-05-05

AI Summary

This work addresses the limitations of conventional multi-example in-context learning, which employs a fixed number of demonstrations and struggles to adapt to varying query difficulty while incurring high computational and memory costs due to long contexts. The authors propose a novel approach that dynamically optimizes the number of examples and efficiently reuses key-value (KV) cache. Specifically, they introduce an entropy-based probing mechanism to adaptively select the optimal number of demonstrations and develop a semantic-aware KV cache reuse strategy that leverages position encoding decoupling and recoding to enable flexible rearrangement of cached key-value pairs. This method uniquely unifies adaptive example selection with semantic-aware cache reuse, achieving both strong performance and significant efficiency gains—outperforming the current state-of-the-art DBSA by approximately 10% in average accuracy while accelerating inference by 4.64×.

Abstract

Many-Shot In-Context Learning (ICL) has emerged as a promising paradigm, leveraging extensive examples to unlock the reasoning potential of Large Language Models (LLMs). However, existing methods typically rely on a predetermined, fixed number of shots. This static approach often fails to adapt to the varying difficulty of different queries, leading to either insufficient context or interference from noise. Furthermore, the prohibitive computational and memory costs of long contexts severely limit Many-Shot's feasibility. To address the above limitations, we propose AdapShot, which dynamically optimizes shot counts and leverages KV cache reuse for efficient inference. Specifically, we design a probe-based evaluation mechanism that utilizes output entropy to determine the optimal number of shots. To bypass the redundant prefilling computation during both the probing and inference phases, we incorporate a semantics-aware KV cache reuse strategy. Within this reuse strategy, to address positional encoding incompatibilities, we introduce a decoupling and re-encoding method that enables the flexible reordering of cached key-value pairs. Extensive experiments demonstrate that AdapShot achieves an average performance gain of around 10% and a 4.64x speedup compared to state-of-the-art DBSA.

Problem

• Many-Shot In-Context Learning
• adaptive shot selection
• KV cache reuse
• computational efficiency
• context length

Innovation

• Adaptive In-Context Learning
• KV Cache Reuse
• Semantic-Aware Caching
• Dynamic Shot Selection
• Positional Encoding Decoupling