Continuous Semantic Caching for Low-Cost LLM Serving

2026-04-21

AI Summary

Existing semantic caching approaches are constrained by discrete and finite query spaces, rendering them inefficient for handling the infinite and continuous semantic queries prevalent in large language model applications. This work addresses this limitation by extending semantic caching to continuous spaces for the first time, introducing a theoretical framework that integrates dynamic ε-net discretization with kernel ridge regression to generalize query cost estimation and optimize caching policies. Building upon this foundation, the authors design an online adaptive algorithm that explicitly accounts for switching costs and prove it achieves sublinear regret. Experimental results demonstrate that the proposed method effectively approximates the continuous optimal policy, significantly reducing both computational and switching overhead while substantially outperforming existing discrete-space approaches.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

As Large Language Models (LLMs) become increasingly popular, caching responses so that they can be reused by users with semantically similar queries has become a vital strategy for reducing inference costs and latency. Existing caching frameworks have proposed to decide which query responses to cache by assuming a finite, known universe of discrete queries and learning their serving costs and arrival probabilities. As LLMs' pool of users and queries expands, however, such an assumption becomes increasingly untenable: real-world LLM queries reside in an infinite, continuous embedding space. In this paper, we establish the first rigorous theoretical framework for semantic LLM response caching in continuous query space under uncertainty. To bridge the gap between discrete optimization and continuous representation spaces, we introduce dynamic $ε$-net discretization coupled with Kernel Ridge Regression. This design enables the system to formally quantify estimation uncertainty and generalize partial feedback on LLM query costs across continuous semantic query neighborhoods. We develop both offline learning and online adaptive algorithms optimized to reduce switching costs incurred by changing the cached responses. We prove that our online algorithm achieves a sublinear regret bound against an optimal continuous oracle, which reduces to existing bounds for discrete query models. Extensive empirical evaluations demonstrate that our framework approximates the continuous optimal cache well while also reducing computational and switching overhead compared to existing methods.

Problem

• semantic caching
• large language models
• continuous query space
• inference cost
• uncertainty

Innovation

• continuous semantic caching
• dynamic ε-net discretization
• Kernel Ridge Regression
• sublinear regret
• LLM serving optimization