🤖 AI Summary
To address the limited context capacity and inefficient repeated-prompt inference in small-parameter language models (LLMs), this paper proposes a model-agnostic KV cache reuse mechanism. The method leverages Sentence-BERT to generate sentence embeddings and enables lossless cross-prompt KV state reuse when a new prompt shares an exact prefix with a historical one—achieved via embedding-based retrieval and serialized cache reconstruction. It supports CPU-side cache persistence and hot reloading, ensuring both generality and deployment friendliness. Experiments demonstrate significantly reduced inference latency under prefix-overlap conditions, with negligible degradation in semantic fidelity; when no overlap exists, behavior matches the baseline exactly, confirming robustness and practicality. To our knowledge, this is the first work achieving lossless, cross-prompt KV cache reuse while maintaining full compatibility with the standard Hugging Face Transformers framework.
📝 Abstract
Whether attention key value (KV) states computed for one prompt for a small LLM can be reused to accelerate inference on a new similar prompt, giving an increase to the space to its context memory using an approach called token recycling. Using a standard Hugging Face setup with DialoGPT-medium (a 345M parameter GPT-2 style decoder trained on 147M Reddit exchanges, 2005 to 2017) as the testbed, we build a cache of past activations and get entries by sentence embeddings, then reuse cached past key values when the cached prompt is an exact prefix of the new input. We compare recycled vs. baseline runs on latency and output fidelity, and log reuse depth in tokens. Reproducibility requires no model modifications, cached KVs are serialized to the CPU, reloaded, and supplied to the generate function to continue decoding from the cached prefix. In tests, we observe consistent speedups when prefix overlap exists, with no material degradation in output semantics, and when overlap is absent, behavior matches baseline.