SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL

2026-08-25

AI Summary

本文解决了异步代理强化学习中的长且可变工具使用轨迹成本高的问题，通过改进单流策略优化方法来提高在线学习效率。

Abstract

Group-relative reinforcement learning waits for sibling rollouts of the same prompt, which is costly for long and variable tool-use trajectories. Single-stream Policy Optimization (SPO) removes this dependency with a persistent prompt-level value estimate, but its recipe whitens one advantage per trajectory before optimizing a token-mean actor loss. We show that trajectory centering generally does not center the token-weighted quantity consumed by the actor, and fix the mismatch by standardizing terminal-outcome advantages under the action-token measure. We additionally organize prompt evidence by the policy event that generated it rather than learner receipt order. Across matched runs on ALFWorld at two model scales and on Math-TIR, SPO++ improves online learning efficiency over SPO. A paired ablation identifies action-token-measure normalization as the strongest tested component.

Problem

• Asynchronous Agentic RL
• Prompt-level Value Estimate
• Token-Weighted Quantity

Innovation

• Stream-Aligned Policy Optimization
• action-token measure normalization
• asynchronous agentic RL