Closing the Calibration Gap in Semantic Caching

2026-06-17

AI Summary

This work addresses a critical gap in existing semantic caching systems, which rely on PR-AUC for evaluation while overlooking practical deployability under fixed decision thresholds, leading to a disconnect between model selection and real-world performance. The authors propose a cache-aware evaluation perspective that reframes model selection as a calibration problem rather than a ranking one, introducing two novel metrics: Precision–Cache Hit Ratio AUC and Calibration Retention Rate. These metrics uniquely decompose the performance gap into a recoverable calibration component and an irreducible data-structure component. Through comprehensive analyses grounded in calibration theory, ablation studies, and comparisons between post-hoc calibration and training objectives, the study demonstrates that calibration discrepancies are predominantly governed by the training objective, with post-processing offering only partial mitigation. The proposed metrics substantially improve the accuracy of model selection at deployment time.

Abstract

Semantic caching cuts LLM inference costs by serving a cached response to semantically similar queries. Standard practice evaluates these systems using PR-AUC, a metric that only measures how well scores rank and ignores whether they are usable at a fixed threshold. We show this mismatch leads to systematically poor deployment choices, as models with the highest PR-AUC are often the worst in operation. We introduce Precision-Cache Hit Ratio (P-CHR) AUC, a cache-aware metric that measures precision across cache utilization levels, and Calibration Retention Rate (CRR), which captures how much offline ranking quality survives at deployment. We decompose the operational gap between offline and deployed quality into a recoverable calibration component and an irreducible structural component fixed by the dataset's positive rate. Our experiments show that the calibration gap is governed by the training objective rather than data scale, and post-hoc calibration only partially closes it. Ultimately, model selection for semantic caching is a calibration problem, not a ranking one, and measuring it is the first step to closing the gap.

Problem

• semantic caching
• calibration gap
• model selection
• deployment performance
• evaluation metrics

Innovation

• semantic caching
• calibration gap
• P-CHR AUC
• Calibration Retention Rate
• model selection