KernelFlume: Elastic Core-Attention Scaling for Agentic Long-Context Decoding

2026-06-28

AI Summary

This work addresses the inefficiencies of current large language model serving systems, which rely on whole-instance scaling under bursty long-context requests, resulting in high latency, excessive memory overhead, and fragmented batching. The authors propose a decoupled architecture that separates stable computational pathways—such as projection and feed-forward networks—from the core attention computation, enabling token-range-granular dynamic routing and elastic scaling of attention states through coordinated weight-holding and weightless attention nodes, thereby eliminating the need for full-model replica deployment. By integrating query-prioritized attention scheduling, inter-layer kernel pipelining, UCX-based communication, and a host-visible graph-signal-driven execution mechanism, the system reduces cost per million output tokens by 32% on A6000 and 61% on H100 GPUs, while maintaining p99 per-token latency at 74 ms and 34 ms, respectively; simulations with larger models project cost reductions of 80–85%.

Abstract

LLM serving is increasingly dominated by long and dynamic decode workloads from agents, reasoning models, and extended conversations. When bursty long-context demand exceeds deployed capacity, existing serving systems typically scale out by launching additional serving instances with model replicas. This instance-level elasticity increases KV capacity only by provisioning another full copy of the model, inheriting startup latency, memory overhead, and batch fragmentation. We present KernelFlume, a decode-centric architecture that disaggregates the stable projection/FFN path from core-attention computation: weight nodes execute dense projection/FFN kernels, while weightless attention nodes store token-range KV partitions and scale with request-state demand. To make this separation elastic, KernelFlume maintains a routing table that maps token ranges to attention-node endpoints. It updates routes at token boundaries and uses host-visible graph signals to drive pre-registered UCX endpoint communication outside the captured CUDA Graph. To preserve low per-token latency after disaggregation, KernelFlume combines query-first core-attention dispatch with inter-layer kernel pipelining, overlapping remote attention and communication with local projection/FFN work. On real GPU testbeds (intra-node A6000 and cross-node H100), under a dynamic long-context agentic workload serving Llama-3.1-8B, KernelFlume sustains flat p99 TPOTs of ~74 ms on A6000 and ~34 ms on H100, while lowering cost per million output tokens by up to 32% and 61%, respectively, relative to full-instance elastic scaling with ServerlessLLM, a state-of-the-art instance-startup method. Replaying the same trace at larger model scale in simulation projects a 56--66% cost reduction over ServerlessLLM, widening to 80--85% with cheaper heterogeneous attention-node hardware and persisting into the million-token context range.

Problem

• long-context decoding
• elastic scaling
• LLM serving
• KV cache
• agentic workloads

Innovation

• disaggregated inference
• elastic attention scaling
• KV cache partitioning
• kernel pipelining
• low-latency LLM serving