When Classic Cache Policies Fail: Learning-Augmented Replacement for Semantic Retrieval Buffers

2026-06-30

AI Summary

Traditional caching policies such as LRU and LFU fail in semantic retrieval scenarios due to the absence of temporal locality and skewed access frequency. This work formulates semantic cache management as an online replacement problem with switching costs and introduces SOLAR, a novel framework that determines update时机 through cumulative regret and leverages Bayesian online learning to optimize content selection from implicit feedback. Theoretical analysis establishes that SOLAR achieves a constant competitive ratio (≤3) and a near-optimal regret bound of O(√(KT log T)), thereby overcoming the dependence on cache size and time horizon inherent in conventional approaches. Empirical results demonstrate that SOLAR improves performance by 5%–75% over FIFO under tight cache constraints, reveals for the first time that classical caching strategies systematically underperform FIFO under semantic workloads, and uncovers an inverted U-shaped relationship between retrieval pool size and quality.

Abstract

LLM agents increasingly rely on retrieval buffers to store and reuse past experience, yet the cache management policies governing these buffers remain largely ad-hoc. We formalize this as an online semantic cache replacement problem with switching costs, where items are matched by embedding similarity and hit quality is continuous rather than binary. Through experiments on two datasets from MemoryBench-Full (LoCoMo, DialSim) with 8 replacement policies, we reveal a surprising finding: classic heuristics (LRU, LFU) \emph{consistently underperform} the naive FIFO baseline on semantic workloads, due to the absence of temporal locality and frequency concentration. We propose SOLAR, a learning-augmented framework that derives modification timing from regret accumulation (achieving $\sim$17\% modification rate) and content selection from Bayesian online learning over implicit retrieval feedback. We prove SOLAR achieves a constant competitive ratio $\leq 3$, independent of cache size and horizon (vs.\ $Ω(K)$ for FIFO), and eviction regret $O(\sqrt{KT\log T})$, matching the $Ω(\sqrt{KT})$ lower bound up to logarithmic factors. Experiments demonstrate 5--75\% relative improvement over FIFO at tight cache sizes, with a clearly characterized phase transition at the working set boundary. Synthetic experiments with 5000-item pools further reveal an inverted-U relationship between pool size and retrieval quality, justifying capacity constraints as a retrieval noise phenomenon rather than a storage limitation.

Problem

• semantic cache
• cache replacement
• retrieval buffers
• LLM agents
• online caching

Innovation

• learning-augmented caching
• semantic retrieval buffer
• online cache replacement
• Bayesian online learning
• competitive ratio