SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning

2026-08-24

AI Summary

本文提出SRPO方法，通过自我反思机制将稀疏的结果反馈转化为密集的学习信号，有效提升大型语言模型在长周期推理任务中的表现。

Abstract

Self-reflection is a powerful mechanism for credit assignment in human learning, converting sparse outcome feedback into actionable guidance. However, its potential for post-training Large Language Models (LLMs) remains underexplored. We propose Self-Reflective Policy Optimization (SRPO), a framework that internalizes this capability. SRPO enables LLMs to analyze their own completed trajectories, synthesize errors into concise "reflection patches," and use reflection-conditioned teacher scores on student on-policy rollouts as dense token-level training signals. This process effectively transforms sparse terminal supervision into dense, token-level learning signals without requiring external critics, separate reward models, or larger teacher models. We demonstrate that SRPO achieves state-of-the-art performance across mathematical reasoning and long-horizon agentic benchmarks with exceptional data efficiency. Using a Qwen3-8B base model, SRPO attains 73.3% on AIME'24 using only 8% (0.08x) of the training FLOPs required by scaled supervised fine-tuning, while significantly improving success rates on WebShop (64.7%), ALFWorld (76.8%), and SWE-Bench-Lite (31.2%). Code is available at https://github.com/Galleons2029/SRPO

Problem

• Self-Reflection
• Large Language Models
• Long-Horizon Reasoning
• Sparse Feedback

Innovation

• Self-Reflective Policy Optimization
• Sparse Outcome Feedback
• Dense Token-Level Signals
• Large Language Models
• Data Efficiency