Skip-Connected Policy Optimization for Implicit Advantage

2026-04-09

AI Summary

This work addresses the challenge that dense rewards, under limited sampling budgets, often yield high-variance and sign-inconsistent advantage estimates for early reasoning tokens, leading to inferior performance compared to strategies using only outcome-based rewards. To mitigate this, the authors propose a two-stage reasoning optimization framework: an upstream module learns via a single forward pass guided by dense rewards provided by a downstream module, which employs Group Relative Policy Optimization (GRPO) and incorporates skip connections that concatenate upstream reasoning with the original problem input. This architecture effectively balances the utilization of intermediate reasoning steps with the avoidance of erroneous paths, revealing an “implicit advantage”—where high-quality intermediate reasoning enhances overall performance even when final accuracy is comparable. The method achieves relative improvements of 3.91% and 6.17% on Qwen2.5-Math-7B and Llama-3.2-3B across mathematical reasoning, general reasoning, and code generation tasks.

Technology Category

• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Group Relative Policy Optimization (GRPO) has proven effective in RLVR by using outcome-based rewards. While fine-grained dense rewards can theoretically improve performance, we reveal that under practical sampling budgets, Monte Carlo estimation yields high-variance and sign-inconsistent advantages for early reasoning tokens, paradoxically underperforming outcome-only GRPO. We propose Skip-Connected Optimization (SKPO), which decomposes reasoning into upstream and downstream phases: upstream receives dense rewards from downstream Monte Carlo sampling with single-stream optimization; downstream maintains group-relative optimization, where a skip connection concatenates the upstream segment with the original problem, enabling the model to leverage helpful upstream reasoning while preserving the freedom to bypass flawed reasoning through direct problem access. Experiments demonstrate improvements of 3.91% and 6.17% relative gains over the strongest baselines on Qwen2.5-Math-7B and Llama-3.2-3B respectively across mathematical benchmarks and out-of-domain tasks including general reasoning and code generation. Further analysis reveals an implicit advantage: SKPO generates trajectories with higher intermediate-step quality even when matched for final correctness.

Problem

• Policy Optimization
• Monte Carlo Estimation
• Advantage Estimation
• Reinforcement Learning
• Reasoning Tokens

Innovation

• Skip-Connected Optimization
• Group Relative Policy Optimization
• Dense Reward
• Monte Carlo Estimation
• Implicit Advantage