Marconi: Prefix Caching for the Era of Hybrid LLMs

2024-11-28

arXiv.org

AI Summary

In hybrid large language models (LLMs) combining attention and recurrent layers, prefix caching suffers from severe inefficiency due to in-place state updates in recurrent layers—rendering conventional caches effective only under exact sequence matches and yielding extremely low reuse rates. Method: This paper introduces the first prefix caching system tailored for hybrid LLMs. It innovatively integrates reuse probability prediction with a dynamic admission and eviction policy guided by the computational savings-to-memory overhead ratio. The design further incorporates cache-hotness modeling, classification of sequence reuse patterns, and lightweight state snapshot management. Results: Evaluated across diverse hybrid models and realistic workloads, the system achieves up to a 34.4× improvement in token-level cache hit rate (reaching 71.1%) and reduces first-token generation latency by 617 ms. These gains significantly enhance caching efficiency and inference throughput.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Hybrid models that combine the language modeling capabilities of Attention layers with the efficiency of Recurrent layers (e.g., State Space Models) have gained traction in practically supporting long contexts in Large Language Model serving. Yet, the unique properties of these models complicate the usage of complementary efficiency optimizations such as prefix caching that skip redundant computations across requests. Most notably, their use of in-place state updates for recurrent layers precludes rolling back cache entries for partial sequence overlaps, and instead mandates only exact-match cache hits; the effect is a deluge of (large) cache entries per sequence, most of which yield minimal reuse opportunities. We present Marconi, the first system that supports efficient prefix caching with Hybrid LLMs. Key to Marconi are its novel admission and eviction policies that more judiciously assess potential cache entries based not only on recency, but also on (1) forecasts of their reuse likelihood across a taxonomy of different hit scenarios, and (2) the compute savings that hits deliver relative to memory footprints. Across diverse workloads and Hybrid models, Marconi achieves up to 34.4$ imes$ higher token hit rates (71.1% or 617 ms lower TTFT) compared to state-of-the-art prefix caching systems.

Problem

• Efficient prefix caching for Hybrid LLMs with Attention and Recurrent layers
• Handling in-place state updates that limit partial sequence cache reuse
• Optimizing cache admission and eviction policies for higher token hit rates

Innovation

• Hybrid LLM prefix caching with admission policies
• Forecasts reuse likelihood across hit scenarios
• Optimizes compute savings relative to memory