Category-Aware Semantic Caching for Heterogeneous LLM Workloads

2025-10-29

AI Summary

Existing unified semantic caching strategies for LLM services suffer from poor cache efficiency due to significant heterogeneity across query types (e.g., code vs. conversational queries) in embedding distribution, temporal freshness requirements, and repetition patterns—resulting in low hit rates (5–15%), high remote search latency (30 ms), and unprofitable cache operation. Method: We propose a category-aware semantic caching framework that dynamically adapts similarity thresholds, TTLs, and memory quotas per query category; employs a hybrid in-memory HNSW index with external document storage; and integrates load-adaptive scheduling and heterogeneous storage management. Contribution/Results: Our approach reduces cache miss penalty to 1/15 of baseline, enables economically viable caching even for low-hit-rate categories, theoretically lowers system load by 9–17%, and achieves efficient coverage across the full query load spectrum—the first such result in LLM caching.

Technology Category

• Search and Optimization: Distributed Search
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

LLM serving systems process heterogeneous query workloads where different categories exhibit different characteristics. Code queries cluster densely in embedding space while conversational queries distribute sparsely. Content staleness varies from minutes (stock data) to months (code patterns). Query repetition patterns range from power-law (code) to uniform (conversation), producing long tail cache hit rate distributions: high-repetition categories achieve 40-60% hit rates while low-repetition or volatile categories achieve 5-15% hit rates. Vector databases must exclude the long tail because remote search costs (30ms) require 15--20% hit rates to break even, leaving 20-30% of production traffic uncached. Uniform cache policies compound this problem: fixed thresholds cause false positives in dense spaces and miss valid paraphrases in sparse spaces; fixed TTLs waste memory or serve stale data. This paper presents category-aware semantic caching where similarity thresholds, TTLs, and quotas vary by query category. We present a hybrid architecture separating in-memory HNSW search from external document storage, reducing miss cost from 30ms to 2ms. This reduction makes low-hit-rate categories economically viable (break-even at 3-5% versus 15-20%), enabling cache coverage across the entire workload distribution. Adaptive load-based policies extend this framework to respond to downstream model load, dynamically adjusting thresholds and TTLs to reduce traffic to overloaded models by 9-17% in theoretical projections.

Problem

• Optimizing semantic caching for heterogeneous LLM workloads with varying characteristics
• Addressing inefficiencies of uniform cache policies across diverse query categories
• Reducing cache miss costs to enable economic viability for low-hit-rate categories

Innovation

• Category-aware semantic caching with adaptive thresholds and TTLs
• Hybrid architecture separates HNSW search from document storage
• Load-based policies dynamically adjust caching to reduce model traffic