RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training

2026-08-19

AI Summary

该研究针对多轮次强化学习训练不稳定的问题，提出了一种反向轮次策略优化（RTPO）方法，通过逆序更新策略来稳定训练过程。

Abstract

Training multi-turn agentic workflows with reinforcement learning (RL) enables large language models to perform complex reasoning, use external tools, and conduct iterative search beyond single-turn settings. Yet multi-turn RL training remains highly unstable, often causing severe performance degradation as the number of turns increases. Through theoretical analysis, we identify three tightly coupled sources of instability: rollout-training context mismatch, weak turn-level credit assignment under sparse terminal rewards, and asynchronous policy drift when short and long trajectories are optimized under different policy versions. We show that these issues share a common structural origin in flattened trajectory optimization and address them through a unified reverse-turn formulation. We propose Reverse-Turn Policy Optimization (RTPO), which organizes multi-turn rollouts as sparse reverse trees and performs turn-level policy updates in temporal reverse order, aligning each decision with its downstream continuation. RTPO enables causally consistent turn-level credit assignment and on-policy continuation to control asynchronous drift. We provide theoretical guarantees showing that RTPO eliminates context mismatch and asynchronous drift under the proposed turn-level formulation, reduces credit bias, and converges to recursive optimality. Experiments on multi-turn agentic RL benchmarks show that RTPO improves upon trajectory- and turn-level baselines by 21.50% and 10.76%, respectively, highlighting its potential to support more stable training for tool-using agents.

Problem

• multi-turn agentic workflows
• reinforcement learning
• instability
• context mismatch
• asynchronous policy drift

Innovation

• Reverse-Turn Policy Optimization
• multi-turn agentic workflows
• causally consistent turn-level credit assignment
• asynchronous policy drift