LipCache: A Local Inference Proxy with Certified Caching for Edge Image Classification Service

2026-08-13

AI Summary

This work addresses the challenge of balancing inference efficiency and classification reliability in edge-based image classification under stringent latency and throughput constraints. Existing semantic caching approaches rely on empirical thresholds, which often lead to silent errors near decision boundaries. To overcome this limitation, the authors propose LipCache—a framework that introduces a lightweight GuardNet satisfying Lipschitz continuity, operating alongside an unmodified main model (MainNet). GuardNet maps inputs into a low-dimensional feature space and computes, for each sample, a provably sound reuse radius based on local classification boundaries and the spectral norm of the classifier head. Cache results are reused only when queries fall within this certified ball; otherwise, inference falls back to MainNet. This approach establishes the first per-sample provably correct caching mechanism for image classification, replacing heuristic hit decisions with geometrically certified guarantees that ensure 100% cache consistency. Experiments on CIFAR, Tiny-ImageNet, and SVHN demonstrate up to 1.65× speedup with negligible accuracy loss, and an enhanced GuardNet variant significantly improves hit rates in multi-class scenarios.

Abstract

As edge-side vision services continue to expand toward low-latency, high-throughput scenarios, reducing the inference cost of vision models without sacrificing reliability has become a central concern. Existing semantic caching methods largely rely on empirical similarity thresholds; while such thresholds improve hit rates, they tend to introduce silent misclassifications near decision boundaries. To address this, we propose \texttt{LipCache}, a certified semantic caching framework for image classification. Without modifying the existing deployed main model, \texttt{MainNet}, the framework introduces a lightweight network, \texttt{GuardNet}, that maps inputs into a low-dimensional feature space subject to a Lipschitz constraint. It then computes a per-sample certified reuse radius from the local classification margin and the spectral norm of the classification head. At runtime, a cached result is reused only when the query feature falls inside the certified reuse ball; otherwise, the query falls back to \texttt{MainNet}. Thus, cache hits are transformed from empirical threshold tests into geometric certification decisions with explicit theoretical boundaries. Across standard image classification tasks like CIFAR, Tiny-ImageNet, and SVHN, \texttt{LipCache} achieves a measured speedup of up to $1.65\times$ with limited end-to-end accuracy degradation, while all accepted cache hits satisfy the \texttt{GuardNet}-side certified-consistency condition. Furthermore, an enhanced \texttt{GuardNet} training recipe substantially improves cache hit rates in the Tiny-ImageNet multi-class extension while maintaining a certified-consistency rate of $100\%$. These results demonstrate that per-sample certified reuse can reduce main-model fallback while preserving theoretical consistency, providing a feasible approach to reliable cache-assisted inference at the edge.

Problem

• edge inference
• semantic caching
• certified consistency
• image classification
• reliability

Innovation

• certified caching
• Lipschitz constraint
• edge inference
• semantic caching
• consistent reuse