DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference

2026-02-24

AI Summary

This work addresses a critical bottleneck in multi-turn LLM agent inference under disaggregated architectures, where KV-Cache storage I/O saturates the network bandwidth of the prefill engine while leaving the decode engine underutilized, severely limiting system throughput. To overcome this, the authors propose a dual-path KV-Cache loading mechanism that introduces a novel direct path from storage to the decode engine alongside the conventional storage-to-prefill path. Leveraging RDMA for efficient cache transfer and a global scheduler for dynamic load balancing, this approach is the first to deliver KV-Cache directly to the decode engine, effectively alleviating storage network congestion and significantly improving resource utilization. Experiments on real-world agent workloads demonstrate up to 1.87× higher offline inference throughput and an average 1.96× increase in online serving throughput, all while strictly meeting service-level objectives (SLOs).

Technology Category

• Machine Learning: Hardware-aware ML
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Mechanism Design

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

The performance of multi-turn, agentic LLM inference is increasingly dominated by KV-Cache storage I/O rather than computation. In prevalent disaggregated architectures, loading the massive KV-Cache from external storage creates a fundamental imbalance: storage NICs on prefill engines become bandwidth-saturated, while those on decoding engines remain idle. This asymmetry severely constrains overall system throughput. We present DualPath, an inference system that breaks this bottleneck by introducing dual-path KV-Cache loading. Beyond the traditional storage-to-prefill path, DualPath enables a novel storage-to-decode path, in which the KV-Cache is loaded into decoding engines and then efficiently transferred to prefill engines via RDMA over the compute network. DualPath combines this optimized data path -- which inherently avoids network congestion and avoids interference with latency-critical model execution communications -- with a global scheduler that dynamically balances load across prefill and decode engines. Our evaluation on three models with production agentic workloads demonstrates that DualPath improves offline inference throughput by up to 1.87$\times$ on our in-house inference system. It can also improve online serving throughput by an average factor of 1.96$\times$ without violating SLO.

Problem

• KV-Cache
• storage bandwidth bottleneck
• agentic LLM inference
• disaggregated architecture
• inference throughput

Innovation

• KV-Cache
• DualPath
• RDMA
• disaggregated architecture
• LLM inference