SCOPE-RL: Optimizing Reasoning Paths Before and After Success

2026-07-13

AI Summary

This work addresses the challenge in reinforcement learning where reliance solely on sparse end-task rewards impedes effective optimization of reasoning trajectories, due to the absence of intermediate feedback before task success and the inability to distinguish efficient from redundant paths afterward. To overcome this, the authors propose SCOPE-RL, a two-stage framework that introduces verifiable prefix-decomposition rewards via answer-hidden subproblem chains prior to success, and subsequently employs correctness-gated process-shaped rewards to refine trajectories while preserving the GRPO update mechanism. The approach innovatively integrates pre- and post-success process-level reward signals through adaptive scaffolding and quality-aware process reinforcement learning, complemented by a Step-Quality Evaluation Protocol validated by human experts. Evaluated on Qwen3-8B-Instruct, SCOPE-RL achieves up to an 11.2 percentage point gain in accuracy and reduces reasoning tokens by 27.1%, demonstrating consistent generalization across diverse models and algorithms.

Abstract

Reinforcement learning with verifiable rewards (RLVR) optimizes LLMs using sparse verifiable final-answer rewards. This sparse anchor reliably verifies whether a trajectory succeeds but provides no direct feedback on the reasoning path that produced it. Before success, prerequisite progress on hard problems receives no reward signal; after success, outcome rewards cannot distinguish well-organized correct trajectories from redundant or locally flawed ones. We introduce SCOPE-RL (Scaffolded Chain Optimization with Process Efficiency), a two-stage framework that densifies this anchor while retaining the GRPO update: Adaptive Scaffolded RL adds prefix-decomposed verifiable rewards on answer-hidden sub-question chains before success, and Quality-Aware Process RL applies correctness-gated process-shape rewards to refine correct trajectories after success. An expert-validated Step-Quality Evaluation Protocol evaluates useful-step density, error localization, and token efficiency beyond final-answer accuracy. On Qwen3-8B-Instruct trained on DAPO-Math and Big-Math, SCOPE-RL improves average accuracy by up to 11.2 pp and reduces reasoning tokens by up to 27.1% over outcome-only GRPO; the gains hold under GSPO and on Qwen3-0.6B-Instruct, indicating that reward-signal densification is complementary to policy-update-level RLVR advances. Code and data are available at https://github.com/tokencraft-lab/SCOPE-RL.

Problem

• reinforcement learning
• reasoning paths
• sparse rewards
• verifiable rewards
• large language models

Innovation

• SCOPE-RL
• reward densification
• reasoning path optimization
• process-efficiency reward
• verifiable sub-question chains