RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents

2026-07-06

AI Summary

This work addresses the challenges in multi-turn interactive tasks where sparse outcome rewards lead to slow convergence and low sample efficiency, while dense process rewards often misalign with the true objective, degrading performance. To overcome these issues, the paper proposes Reward-Swap Policy Optimization (RSPO), a novel method featuring a dynamic reward-swapping mechanism that integrates dense process rewards during training to accelerate learning, while strictly aligning the ultimate optimization objective with the true outcome reward. RSPO also enhances policy exploration diversity without compromising goal fidelity. The approach is compatible with mainstream algorithms such as PPO, GRPO, and GiGPO, and demonstrates consistent and significant performance improvements on the WebShop and ALFWorld benchmarks.

Abstract

Reinforcement learning holds significant potential for training large language models (LLMs) to handle multi-turn interactive tasks. However, in long-horizon, multi-turn tasks characterized by sparse outcome rewards, directly training with outcome rewards often results in slow convergence due to the sparsity of signals and the lack of fine-grained feedback. Furthermore, the model may fail to learn successful trajectories that are not sampled during training, thereby limiting its performance. Conversely, while employing customized dense process rewards provides richer signals and accelerates convergence, these surrogate rewards may exhibit potential misalignment with the ground-truth outcome rewards. This inconsistency can bias the training direction and ultimately degrade the model's final performance. In this work, we propose Reward-Swap Policy Optimization (RSPO), a method designed to leverage the rich information from dense process rewards to facilitate training with outcome rewards. By utilizing a reward-swap mechanism, RSPO ensures the diversity of sampled trajectories while guaranteeing consistency between the optimization objective and the true outcome rewards, thereby elevating the performance ceiling of the model. We conduct extensive experiments on two challenging agent benchmarks, WebShop and ALFWorld. By applying our method to various reinforcement learning algorithms, including GRPO, PPO, and GiGPO, we demonstrate that RSPO achieves consistent performance improvements across different baselines and benchmarks.

Problem

• sparse rewards
• dense process rewards
• reward misalignment
• multi-turn LLM agents
• reinforcement learning

Innovation

• Reward-Swap Policy Optimization
• multi-turn LLM agents
• sparse reward
• dense process rewards
• trajectory diversity