Cache-Resident LLM Inference in GB-Scale Last-Level Caches

2026-06-23

AI Summary

This work addresses the inefficiency of frequent memory accesses in large language model (LLM) inference, which underutilizes the substantial last-level cache capacity of modern multi-core CPUs. To overcome this limitation, the authors propose a cache-resident execution model that decouples weight-intensive computations from attention mechanisms and KV cache management, assigning them to dedicated resource domains. By relaxing synchronization constraints based on sub-operator dependencies, the approach breaks conventional operator boundaries through weight cache residency decoupled from KV cache capacity, locality-aware data placement, and a lightweight static runtime. This design significantly reduces coordination overhead, achieving 2.04× to 11.51× per-token inference speedup on Llama-3.2-3B and Llama-2-7B models, with a theoretical peak acceleration of 13.9×.

Abstract

Large language model (LLM) inference is increasingly dominated by data movement across the memory hierarchy. Recent 3D-stacked cache technologies have enabled GB-scale last-level caches in modern server CPUs, making it possible to keep reusable model weights on chip and exploit cache bandwidth and latency. Achieving this regime is not straightforward: deeper pipelining for weight residency increases in-flight requests and KV-cache footprint, while cache-resident operators make operator-boundary synchronization a visible bottleneck. We present a cache-resident execution model for inference on hierarchical-memory clustered systems. The model separates weight-centric operators from attention and KV-cache management into dedicated resource domains, keeping reusable weights cache-resident while scaling KV capacity independently of pipeline depth. It also relaxes synchronization from operator boundaries to true sub-operator dependencies, reducing coordination overhead in the cache-resident regime. We instantiate this model on a multi-socket CPU cluster with a weight-attention decoupled architecture, locality-aware placement, and a specialized static runtime. The prototype substantially outperforms equally provisioned llama.cpp. On deployed Llama-3.2-3B and Llama-2-7B configurations, it achieves 2.04x-11.51x speedup on time-per-output-token (TPOT). Under a validated analytical model, it further reaches up to 13.9x TPOT speedup across model sizes, context lengths, and batch sizes. These results show that commodity CPUs with GB-scale last-level caches can support efficient LLM inference when execution is organized around cache residency, decoupled state management, and dependency-aware coordination.

Problem

• LLM inference
• cache residency
• memory hierarchy
• KV-cache
• synchronization bottleneck

Innovation

• cache-resident inference
• weight-attention decoupling
• KV-cache scalability
• sub-operator synchronization
• hierarchical-memory LLM execution