TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks

2026-05-16

AI Summary

Existing KV cache quantization methods struggle to capture the heterogeneous importance structure along the three axes of time, modality, and semantic role in agent reasoning, leading to significant performance degradation. This work proposes TriAxialKV, the first mixed-precision quantization framework that jointly models this triaxial heterogeneity: each token is assigned a triaxial label, and bitwidths (INT2/INT4) are dynamically allocated based on sensitivity-aware calibration. The approach further integrates a custom Triton-fused decoding kernel and an end-to-end serving system. Evaluated on the Qwen3-VL-32B-Thinking agent, TriAxialKV achieves near-BF16 accuracy with INT2/INT4 quantization, enabling a 4.5× expansion in KV cache capacity and a 30% improvement in end-to-end throughput.

Abstract

Agentic workloads have emerged as a major workload for LLM inference. They differ significantly from chat-only workloads, requiring long-context processing, the ability to handle multimodal inputs, and structured multi-turn interactions with tool calling capabilities. As a result, their context exhibits structure that can carry different importance along three key axes: temporal recency to the current turn, modality such as text or image tokens, and semantic role such as user queries, tool calls, observations, or reasoning. These axes capture distinct token behaviors and lead to different sensitivities to KV-cache compression. However, existing KV-cache quantization methods are typically homogeneous or exploit only heterogeneity on a single dimension, such as temporal proximity or modality, overlooking the interactions among them. To this end, we introduce TriAxialKV, a novel mixed-precision KV-cache quantization scheme that assigns each token a triaxial tag, calibrates per-tag sensitivity, and allocates INT2/INT4 bitwidths under a fixed memory budget. We implement TriAxialKV as an end-to-end serving system, comprising calibration, mixed-precision quantization and memory management, and custom fused Triton decode kernels. When using Qwen3-VL-32B-Thinking as a computer-use agent operating the OSWorld, TriAxialKV matches the accuracy of SGLang with BF16 KV cache while supporting 4.5$\times$ KV cache size and achieving 30% higher end-to-end throughput, when running on real GPU systems.

Problem

• KV-cache quantization
• agentic inference
• low-precision
• heterogeneous compression
• structured context

Innovation

• KV-cache quantization
• mixed-precision
• agentic inference
• triaxial tagging
• low-bitwidth