Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning

2026-05-26

AI Summary

This work addresses the challenge in conventional multi-agent reinforcement learning where trajectory-level rewards hinder accurate credit assignment to effective actions within failed episodes. To overcome this limitation, the authors propose a step-level credit assignment method based on a unified state-transition graph. By leveraging graph embeddings and global structural information, the approach estimates the distance from each state to the goal and dynamically allocates credit to individual actions accordingly. This is the first method to incorporate graph-structured representations into multi-agent policy optimization, enabling fine-grained evaluation of how closely agents progress toward task objectives. Experimental results demonstrate that the proposed framework significantly improves training efficiency and achieves state-of-the-art performance across multiple complex multi-agent benchmark tasks.

Abstract

Group-based reinforcement learning (RL) methods have achieved remarkable success in improving the performance of large language models (LLMs) and have been rapidly extended to agentic tasks. However, their credit assignment relies heavily on coarse-grained trajectory-level attribution according to final outcomes, making it difficult to capture the contribution of individual steps, such as valuable steps obscured within failed trajectories. To uncover latent information and enable more faithful step-level credit assignment, we propose Graph-based Group Policy Optimization (GraphGPO), which first aggregates all rollout trajectories into a unified state-transition graph and then estimates the distance from each state to the task goal using the global information encoded in the graph. Finally, GraphGPO assigns credit to each edge by estimating a graph-based advantage, based on how much the transition reduces the distance to the task goal. In this way, GraphGPO significantly improves training efficiency and achieves state-of-the-art performance across a range of challenging benchmarks.

Problem

• credit assignment
• trajectory-level attribution
• step-level contribution
• reinforcement learning
• agentic tasks

Innovation

• graph-based credit assignment
• step-level attribution
• state-transition graph
• agentic reinforcement learning
• GraphGPO