SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks

2026-04-10

AI Summary

Standard token-level PPO struggles in long-horizon reasoning tasks due to unstable temporal credit assignment and the high memory overhead of value models, while critic-free approaches alleviate these issues at the cost of substantial computational expense. This work proposes Sequence-level PPO (SPPO), which formulates reasoning as a sequence-level contextual bandit problem and employs a decoupled scalar value function to produce low-variance advantage estimates. SPPO achieves both sample efficiency and update stability without requiring multi-sample baselines. Experimental results demonstrate that SPPO significantly outperforms standard PPO on mathematical reasoning benchmarks, matching the performance of computationally intensive group-based policy methods while substantially improving training throughput and resource efficiency.

Technology Category

• Reasoning under Uncertainty: Sequential Decision Making
• Machine Learning: Online Learning & Bandits
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Proximal Policy Optimization (PPO) is central to aligning Large Language Models (LLMs) in reasoning tasks with verifiable rewards. However, standard token-level PPO struggles in this setting due to the instability of temporal credit assignment over long Chain-of-Thought (CoT) horizons and the prohibitive memory cost of the value model. While critic-free alternatives like GRPO mitigate these issues, they incur significant computational overhead by requiring multiple samples for baseline estimation, severely limiting training throughput. In this paper, we introduce Sequence-Level PPO (SPPO), a scalable algorithm that harmonizes the sample efficiency of PPO with the stability of outcome-based updates. SPPO reformulates the reasoning process as a Sequence-Level Contextual Bandit problem, employing a decoupled scalar value function to derive low-variance advantage signals without multi-sampling. Extensive experiments on mathematical benchmarks demonstrate that SPPO significantly surpasses standard PPO and matches the performance of computation-heavy group-based methods, offering a resource-efficient framework for aligning reasoning LLMs.

Problem

• Long-Horizon Reasoning
• Proximal Policy Optimization
• Chain-of-Thought
• Credit Assignment
• Sample Efficiency

Innovation

• Sequence-Level PPO
• Contextual Bandit
• Low-Variance Advantage
• Long-Horizon Reasoning
• Sample Efficiency