AgentV-RL: Scaling Reward Modeling with Agentic Verifier

2026-04-17

AI Summary

Traditional verifiers are prone to unreliable evaluation due to error propagation in intermediate reasoning steps and insufficient access to external knowledge. To address this, this work proposes the Agentic Verifier framework, which introduces a novel forward–backward agent collaboration mechanism. It reformulates reward modeling as a multi-turn, tool-augmented bidirectional reasoning process, enabling active exploration and dynamic tool invocation. This approach yields interpretable and highly reliable verification outcomes. Notably, with only a 4B-parameter model under test-time scaling settings, the method outperforms the current state-of-the-art outcome reward model (ORM) by 25.2%, achieving substantially improved verification accuracy on complex reasoning tasks.

Technology Category

• Multiagent Systems: Mechanism Design
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Knowledge Representation and Reasoning: Argumentation

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Verifiers have been demonstrated to enhance LLM reasoning via test-time scaling (TTS). Yet, they face significant challenges in complex domains. Error propagation from incorrect intermediate reasoning can lead to false positives for seemingly plausible solutions, while lacking external grounding makes verifiers unreliable on computation or knowledge-intensive tasks. To address these challenges, we propose Agentic Verifier, a framework that transforms reward modeling into a multi-turn, tool-augmented deliberative process. We introduce complementary forward and backward agents: one traces solutions from premises to conclusions, while the other re-checks conclusions against their underlying premises. This bidirectional process enables a comprehensive, reliable, and interpretable assessment of solutions. To facilitate practical deployment, we propose AgentV-RL. Through proactive exploration and reinforcement learning, the verifier autonomously interleaves tool-use with internal reasoning. Extensive experiments show that Agentic Verifier yields consistent performance gains under both parallel and sequential TTS. Notably, our 4B variant surpasses state-of-the-art ORMs by 25.2%, positioning it as a promising paradigm for agentic reward modeling.

Problem

• verifier
• error propagation
• external grounding
• reward modeling
• complex reasoning

Innovation

• Agentic Verifier
• Reward Modeling
• Tool-Augmented Reasoning
• Bidirectional Verification
• Reinforcement Learning