HYPIC: Accelerating Hybrid-Attention LLM Serving with Position-Independent Caching

2026-07-01

AI Summary

This work addresses the high prefill computation overhead caused by concatenated long contexts in retrieval-augmented generation and agentic large language models, as well as the incompatibility of existing position-agnostic caching with hybrid attention mechanisms. We propose the first position-agnostic caching method supporting hybrid attention, which preserves segment self-containment through a segment-wise cumulative shift operator and a boundary-stitching recomputation mechanism that only recomputes small cross-segment windows to restore attention. Additionally, we introduce segment-level state composition algebraic primitives tailored for linear attention, enabling parallel prefill for cache-miss requests. Experiments across four hybrid attention models and five workload types show an average 2.45× reduction in time-to-first-token latency, up to 2.0× higher peak throughput, and no more than a 3.3 percentage point drop in accuracy.

Abstract

In retrieval augmented generation (RAG) and agentic LLM serving, prompts are assembled from independent segments into long contexts, making the prefill stage dominate the per-request computation cost. To this cost, two directions have emerged in parallel: position-independent caching (PIC) admits KV reuse for non-contiguous segments shared across different requests, while hybrid-attention models reduce computation complexity by replacing most full-attention layers with linear attention. However, they cannot coexist: applying PIC to hybrid-attention models breaks down because per-token KV-cache reuse primitives do not transfer to the per-request recurrent state. In this work, we present Hypic, the first serving system for hybrid-attention LLMs with position-independent caching. For linear-attention layers, we identify the segment-cumulative transition operator as the missing algebraic primitive, and cache it alongside each segment's zero-start end-state, enabling near-exact and constant-time state composition of independently cached segments. For the remaining full-attention layers, existing PIC methods also fail as linear layers do not expose the per-token hidden states for selective recomputation. We show that the most significant attention deviation concentrates at segment boundaries, so recomputing only a small seam window at each boundary suffices to restore cross-segment lookback. Finally, Hypic exploits segment-level self-containment to parallelize cache-miss prefill across instances, turning long cold requests -- a major tail-latency contributor under both prefix caching and prior PIC -- into an accelerable workload. Evaluated across four hybrid-attention models and five workloads, Hypic reduces time-to-first-token (TTFT) by 2.45x on average and improves peak throughput by up to 2.0x over existing systems, while staying within 3.3 points of full-recompute accuracy.

Problem

• retrieval augmented generation
• hybrid-attention LLMs
• position-independent caching
• prefill acceleration
• KV-cache reuse

Innovation

• position-independent caching
• hybrid-attention
• linear attention
• KV-cache reuse
• segment-level parallelism