Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning

2026-07-05

AI Summary

This work addresses the challenge in group-based reinforcement learning where step-level advantage estimation struggles to balance fairness and coverage in long-horizon tasks. The authors propose ProGPO, a method that maintains action history consistency while leveraging state potentials to derive transferable credit signals that augment sparse intra-group comparisons, enabling context-consistent step-level policy optimization. Its key innovation lies in reliably estimating state potentials without requiring a learned critic, instead integrating exact prefix-action contrasts, semantic expansion, and inverse-variance fusion across historical depths. Experimental results demonstrate that ProGPO significantly outperforms existing RL baselines on ALFWorld and WebShop benchmarks and exhibits strong scalability when applied to the Qwen2.5-3B-Instruct model.

Abstract

Group-based reinforcement learning (RL) has become an effective paradigm for improving large language model agents on long-horizon interactive tasks. To obtain finer-grained policy updates than trajectory-level optimization, recent work has moved toward step-level group-based RL, where intermediate steps are grouped and compared within a rollout batch. However, step-level advantage estimation is sensitive to how groups are formed: grouping by broad state keys improves coverage but may compare actions taken under different histories, while enforcing historical consistency yields fairer comparisons at the cost of fragmented groups and missing peer-comparison signal. In this paper, we propose ProGPO (Progress- and Reliability-Oriented Group Policy Optimization), a learned-critic-free method for context-consistent step-level learning. ProGPO keeps exact-prefix action comparison, and complements sparse peer comparisons with transition credit derived from rollout-based state potentials. To estimate these potentials reliably, ProGPO combines semantic expansion with inverse-variance fusion across history depths. We evaluate ProGPO on two challenging agentic tasks, ALFWorld and WebShop, with Qwen2.5-1.5B-Instruct. Results show that ProGPO improves over matched agentic RL baselines under comparable computational overhead, and additional Qwen2.5-3B-Instruct experiments further test the scalability of the proposed method.

Problem

• group-based reinforcement learning
• step-level optimization
• historical consistency
• peer comparison
• advantage estimation

Innovation

• Group-based Reinforcement Learning
• Step-level Policy Optimization
• Context-consistent Comparison
• State Potentials
• Inverse-variance Fusion