Process Reward Agents for Steering Knowledge-Intensive Reasoning

2026-04-10

AI Summary

This work addresses the challenge in knowledge-intensive reasoning where intermediate steps are difficult to verify locally, often leading to error propagation. To mitigate this, the authors propose a test-time dynamic reasoning approach that freezes the policy model and integrates retrieval augmentation with an online process reward mechanism. At each reasoning step, candidate trajectories are scored and pruned based on process-level rewards, enabling the selection of high-quality reasoning paths without requiring model updates. This method pioneers the incorporation of a process reward model into a dynamic search decoding framework, substantially enhancing the reasoning capabilities of frozen models across scales. On MedQA, Qwen3-4B achieves 80.8% accuracy—a new state-of-the-art for 4B-scale models—and yields an average improvement of 25.7% across model sizes ranging from 0.5B to 8B.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Search and Optimization: Metareasoning and Metaheuristics
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Reasoning in knowledge-intensive domains remains challenging as intermediate steps are often not locally verifiable: unlike math or code, evaluating step correctness may require synthesizing clues across large external knowledge sources. As a result, subtle errors can propagate through reasoning traces, potentially never to be detected. Prior work has proposed process reward models (PRMs), including retrieval-augmented variants, but these methods operate post hoc, scoring completed trajectories, which prevents their integration into dynamic inference procedures. Here, we introduce Process Reward Agents (PRA), a test-time method for providing domain-grounded, online, step-wise rewards to a frozen policy. In contrast to prior retrieval-augmented PRMs, PRA enables search-based decoding to rank and prune candidate trajectories at every generation step. Experiments on multiple medical reasoning benchmarks demonstrate that PRA consistently outperforms strong baselines, achieving 80.8% accuracy on MedQA with Qwen3-4B, a new state of the art at the 4B scale. Importantly, PRA generalizes to unseen frozen policy models ranging from 0.5B to 8B parameters, improving their accuracy by up to 25.7% without any policy model updates. More broadly, PRA suggests a paradigm in which frozen reasoners are decoupled from domain-specific reward modules, allowing the deployment of new backbones in complex domains without retraining.

Problem

• knowledge-intensive reasoning
• process reward
• reasoning verification
• error propagation
• external knowledge

Innovation

• Process Reward Agents
• online step-wise reward
• retrieval-augmented reasoning
• frozen policy steering
• search-based decoding