ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cache

2026-04-07

AI Summary

This work addresses the severe GPU memory bottleneck and throughput degradation in multi-LoRA agent collaborative inference caused by divergent key-value (KV) cache states. To tackle this, the authors introduce— for the first time in multi-LoRA serving—an operating system–level fork with copy-on-write (CoW) mechanism, coupled with a novel DualRadixTree cache organization and a specialized ResidualAttention kernel. This design physically decouples shared context from agent-specific KV caches and enables their efficient reconstruction. By leveraging the activation sparsity inherent in LoRA adapters, the proposed approach achieves up to 3.0× higher throughput across various large language models and tasks, with negligible degradation in generation quality.

Abstract

The serving paradigm of large language models (LLMs) is rapidly shifting towards complex multi-agent workflows where specialized agents collaborate over massive shared contexts. While Low-Rank Adaptation (LoRA) enables the efficient co-hosting of these specialized agents on a single base model, it introduces a critical memory footprint bottleneck during serving. Specifically, unique LoRA activations cause Key-Value (KV) cache divergence across agents, rendering traditional prefix caching ineffective for shared contexts. This forces redundant KV cache maintenance, rapidly saturating GPU capacity and degrading throughput. To address this challenge, we introduce ForkKV, a serving system for multi-LoRA agent workflows centered around a novel memory management paradigm in OS: fork with copy-on-write (CoW). By exploiting the structural properties of LoRA, ForkKV physically decouples the KV cache into a massive shared component (analogous to the parent process's memory pages) and lightweight agent-specific components (the child process's pages). To support this mechanism, we propose a DualRadixTree architecture that allows newly forked agents to inherit the massive shared cache and apply CoW semantics for their lightweight unique cache. Furthermore, to guarantee efficient execution, we design ResidualAttention, a specialized kernel that reconstructs the disaggregated KV cache directly within on-chip SRAM. Comprehensive evaluations across diverse language models and practical datasets of different tasks demonstrate that ForkKV achieves up to 3.0x the throughput of state-of-the-art multi-LoRA serving systems with a negligible impact on generation quality.

Problem

• multi-LoRA
• KV cache divergence
• shared context
• memory bottleneck
• LLM serving

Innovation

• Copy-on-Write
• KV Cache Disaggregation
• Multi-LoRA Serving
• DualRadixTree
• ResidualAttention