GraphPO: Graph-based Policy Optimization for Reasoning Models

2026-06-17

AI Summary

Traditional reinforcement learning struggles with inefficient exploration due to redundant intermediate steps and insufficient process supervision caused by sparse rewards in reasoning model optimization. This work proposes a novel policy optimization framework based on directed acyclic graphs (DAGs), which, for the first time, models reasoning trajectories as graph structures. By clustering semantically equivalent states and merging equivalent path suffixes, the framework reduces redundancy and introduces an edge-level advantage allocation mechanism that lowers variance in advantage estimation while enhancing both exploration efficiency and diversity. Inspired by graph neural networks, the approach enables effective process supervision. Evaluated across three large language models and multiple reasoning and agent-based search benchmarks, the method significantly outperforms chain- and tree-based baselines under identical computational budgets.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has become a standard paradigm for enhancing the capability of large reasoning models. RLVR typically samples responses independently and optimizes the policy using from final answers. This paradigm has two limitations. First, independently responses often contain similar intermediate reasoning steps, causing redundant exploration and wasted computation. Second, sparse final-answer rewards make it hard to identify useful steps. Tree-based methods partly address this problem by sharing prefixes and comparing branches from the same prefix to provide fine-grained signals. However, tree branches are still expanded independently. When different branches reach similar reasoning states, they cannot share information and repeat similar exploration. Moreover, tree-based methods ignore such dispersion and only perform local comparisons within separate branches, which can lead to higher variance in advantage estimation. To address this challenge, we propose GraphPO (Graph-based Policy Optimization), a novel RL framework that represents rollouts as a directed acyclic graph, with reasoning steps as edges and semantic states summarized from the reasoning paths as nodes. GraphPO merges semantically equivalent reasoning paths into equivalence classes, allowing them to share suffixes and reallocating budget away from redundant expansions to diverse exploration. Furthermore, we assign efficiency advantages to incoming edges and correctness advantages to outgoing edges, thereby improving inference efficiency while deriving process supervision from outcome. Theory shows that GraphPO reduces advantage-estimation variance and enhances reasoning efficiency. Experiments on three LLMs across reasoning and agentic search benchmarks show that GraphPO consistently outperforms chain- and tree-based baselines with the same token budgets or response budgets.

Problem

• redundant exploration
• sparse rewards
• reasoning efficiency
• advantage estimation variance
• intermediate reasoning steps

Innovation

• Graph-based Policy Optimization
• Directed Acyclic Graph
• Semantic State Merging
• Advantage Estimation Variance Reduction
• Efficiency-Correctness Dual Advantage