KV-PRM: Efficient Process Reward Modeling via KV-Cache Transfer for Multi-Agent Test-Time Scaling

2026-07-10

AI Summary

This work addresses the quadratic computational overhead of conventional text-based reward models in long-context multi-agent reasoning, which stems from repeatedly encoding reasoning trajectories and hinders scalability. The authors propose KV-PRM, the first method to formally demonstrate that the information capacity of key-value (KV) caches in large language model inference strictly exceeds that of raw input text. Leveraging this insight, KV-PRM evaluates trajectory quality directly from a single validation token using transferred KV cache representations, thereby eliminating redundant re-encoding. Integrated with test-time scaling and multi-agent coordination, the approach matches or surpasses text-based PRMs on MATH, GSM8K, and AIME benchmarks while reducing scoring FLOPs by 5,000×, latency by 37×, and memory usage by 34×—substantially overcoming the computational bottlenecks of traditional reward modeling.

Abstract

Process Reward Models (PRMs) have been proven to be highly effective in guiding test-time scaling (TTS) methods, which significantly boost the capabilities of LLM-based multi-agent systems. However, existing PRMs are text-based: they re-encode the entire trajectory text from scratch. In long multi-agent rollouts, the scoring cost, growing quadratically with respect to sequence length L, creates a severe computational bottleneck, severely limiting PRMs' application in long-context scenarios. To resolve this, we introduce KV-PRM, a highly efficient process reward model that eliminates the heavy text re-encoding by directly reading the KV cache produced naturally during the LLM's generation phase. By processing a single "verify token" against the pre-existing KV cache, KV-PRM reduces the scoring cost from O(L^2) to O(L). We formally prove that the KV cache contains strictly greater information capacity than text, and is more efficient for downstream reward modeling. Empirically, across the MATH, GSM8K, and AIME benchmarks, KV-PRM matches or strictly outperforms text-PRMs under various TTS methods such as Beam Search, MCTS, and Weighted Voting, with up to a 5,000x reduction in scoring FLOPs, a 37x reduction in latency, and a 34x reduction in per-sequence memory footprint compared to text-based PRMs.

Problem

• Process Reward Models
• Test-Time Scaling
• KV-Cache
• Multi-Agent Systems
• Computational Efficiency

Innovation

• KV-Cache Transfer
• Process Reward Modeling
• Test-Time Scaling
• Multi-Agent Systems
• Efficient Inference