ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse

2026-05-16

AI Summary

This work addresses the challenge of KV cache overflow in large language model inference, where shared prefixes often exceed local memory capacity, and existing remote DRAM solutions suffer from high cost and poor scalability. The paper presents the first system that leverages S3-compatible object storage for KV cache reuse, co-designing the storage protocol and transfer scheduling to fetch layer-wise data in alignment with GPU consumption order. By integrating concurrent request scheduling with a bandwidth-aware mechanism, the system effectively overlaps computation and data transfer. Built atop RoCE networking, the NIXL inference library, Ceph RGW, and DAOS, the approach incurs only 5.6% higher latency than local DRAM at 64K context length and adds 56–75 ms at 4K context. Under shared bandwidth conditions, it reduces time-to-first-token (TTFT) by 1.2–1.8× compared to equal-bandwidth allocation schemes.

Abstract

Prefix KV caching has become a key mechanism in LLM serving: it reduces time to first token (TTFT) by avoiding redundant computation across requests that share a prefix (i.e., the system prompt). However, the accumulated KV cache is often larger than what GPU memory and local DRAM can hold. To preserve latency, current systems keep the KV cache in remote DRAM pools, increasing serving-cluster size and cost. In this paper, we explore a different approach: storing the KV cache in S3-compatible object storage so that capacity is no longer the constraint, while minimizing the impact on TTFT. We propose ObjectCache, which co-designs the storage protocol and transfer schedule so that the storage server delivers KV cache data in the order the GPU consumes it, overlapping data transfer with compute across concurrent requests. We prototype ObjectCache on a 100 Gbps RoCE cluster with NIXL (an inference library that abstracts storage and memory), Ceph RGW (an Object Gateway for clusters), and DAOS (an open source storage system). For 64K contexts, common in today's systems, ObjectCache adds only 5.6\% latency over local DRAM; for 4K contexts, where less compute is available to mask transfer, ObjectCache adds 56--75\,ms over the optimal local layerwise baseline. Under shared bandwidth caps, our scheduler reduces added TTFT by 1.2--1.8x compared with equal bandwidth sharing.

Problem

• KV cache
• LLM serving
• object storage
• latency
• memory capacity

Innovation

• ObjectCache
• KV cache reuse
• layerwise retrieval
• object storage
• LLM inference