PGPO: Potential-Guided Policy Optimization for Multi-Turn Agentic Tasks

2026-09-02

AI Summary

为解决多轮代理任务中稀疏终端奖励导致的中间动作信用分配不精细问题，提出基于势能引导的策略优化方法PGPO，通过估计状态势能并传播跨轨迹信用。

Abstract

Group-based reinforcement learning (RL) has become an effective paradigm for LLM post-training, but in multi-turn agentic tasks with sparse terminal rewards, it often provides coarse credit for intermediate actions. To obtain more fine-grained credit assignment, recent work such as GiGPO introduces step-level advantages for intermediate actions. However, these step-level signals still rely on the final outcome of each individual trajectory. As a result, actions within failed trajectories can remain poorly differentiated, so effective actions can receive the same unfavorable credit as erroneous ones. In this work, we propose Potential-Guided Policy Optimization (PGPO) for multi-turn agentic tasks. PGPO estimates empirical state potentials from anchor-state-group return statistics within each rollout group. It then derives action advantages from potential differences between adjacent states, enabling cross-trajectory credit propagation. This provides finer-grained step-level credit assignment, especially within failed trajectories. Experiments on ALFWorld and WebShop show strong overall performance relative to recent group-based RL methods. Further analysis provides evidence that PGPO yields more informative failure-side credit signals with negligible training overhead.

Problem

• reinforcement learning
• multi-turn agentic tasks
• sparse terminal rewards
• credit assignment

Innovation

• Potential-Guided Policy Optimization
• empirical state potentials
• cross-trajectory credit propagation
• multi-turn agentic tasks