PVPO: Pre-Estimated Value-Based Policy Optimization for Agentic Reasoning

2025-08-28

AI Summary

Existing critic-free reinforcement learning approaches (e.g., group-wise policy optimization) rely on multiple intra-group rollouts for advantage estimation, suffering from high computational cost and susceptibility to local optima. This paper proposes PVPO: a novel policy optimization framework that introduces a reference model to perform pre-rollouts, establishing value anchors that correct cumulative bias in intra-group comparisons; it further integrates data pre-sampling with difficulty-aware sample selection to reduce reliance on real-time rollouts. Operating entirely within a critic-free paradigm, PVPO enables more stable and efficient advantage estimation. Evaluated across nine benchmark datasets spanning two domains, PVPO achieves state-of-the-art performance, significantly improving training stability, generalization, and scalability to large language models. The method establishes a scalable, low-variance paradigm for policy optimization.

Technology Category

• Search and Optimization: Learning to Search
• Computer Vision: Diffusion Models for Vision
• Multiagent Systems: Adversarial Agents

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Critic-free reinforcement learning methods, particularly group policies, have attracted considerable attention for their efficiency in complex tasks. However, these methods rely heavily on multiple sampling and comparisons within the policy to estimate advantage, which may cause the policy to fall into local optimum and increase computational cost. To address these issues, we propose PVPO, an efficient reinforcement learning method enhanced by an advantage reference anchor and data pre-sampling. Specifically, we use the reference model to rollout in advance and employ the calculated reward score as a reference anchor. Our approach effectively corrects the cumulative bias introduced by intra-group comparisons and significantly reduces reliance on the number of rollouts. Meanwhile, the reference model can assess sample difficulty during data pre-sampling, enabling effective selection of high-gain data to improve training efficiency. Experiments conducted on nine datasets across two domains demonstrate that PVPO achieves State-Of-The-Art (SOTA) performance. Our approach not only demonstrates robust generalization across multiple tasks, but also exhibits scalable performance across models of varying scales.

Problem

• Reduces reliance on multiple sampling for advantage estimation
• Addresses local optimum and high computational cost issues
• Improves training efficiency through data pre-sampling technique

Innovation

• Advantage reference anchor for bias correction
• Data pre-sampling for high-gain selection
• Reduced rollout reliance through pre-estimation