TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning

2026-06-30

AI Summary

This work addresses the limitation of conventional reinforcement learning methods that rely solely on sparse final rewards for credit assignment, often penalizing effective exploration or rewarding redundant actions. The authors propose TRIAGE, a framework that classifies action segments into four semantic roles—critical progress, beneficial exploration, no progress, or regression—and uses these labels to generate bounded procedural rewards. These intermediate signals are combined with the final task reward to refine policy gradients. Theoretically, TRIAGE achieves optimal segment-level correction using only role labels, substantially reducing advantage estimation error and policy gradient variance. Empirically, it improves task success rates on ALFWorld, Search-QA, and WebShop, while reducing interaction steps by 10.4%–14.8% compared to GRPO, outperforming both scalar procedural rewards and shared-backbone value baselines.

Abstract

Agentic reinforcement learning requires assigning credit to environment-facing actions such as searches, clicks, edits, navigation commands, and object interactions. Standard GRPO uses the final verifier outcome as a uniform advantage over all action tokens. This outcome signal is useful but structurally incomplete: it punishes useful exploration in failed rollouts and reinforces redundant or regressive actions in successful rollouts. We propose TRIAGE, a role-typed credit assignment framework that adds a semantic role axis to outcome credit. A structured judge classifies each segment as decisive progress, useful exploration, no-progress infrastructure, or regression, and a fixed role-conditioned rule maps these labels to bounded segment-level process rewards. This keeps verifier outcomes as the source of optimization direction while correcting the two main blind spots of outcome-only credit. We further show that role-conditioned credit is the optimal segment-level correction expressible from role labels alone -- a projection of the per-segment advantage residual onto the role variable -- so that the fixed role constants reduce advantage estimation error whenever the judge is reliable, and we connect this to lower-variance policy gradients. Across ALFWorld, Search-QA, and WebShop, TRIAGE improves success rates over GRPO for two policy models and outperforms both a scalar judge-derived process reward and an outcome-supervised shared-backbone value baseline. Ablations show that the gain comes from role typing rather than merely adding dense rewards: reliable detection of regression inside successful trajectories is the dominant contributor, while exploration credit provides a consistent secondary gain; on completed ALFWorld and WebShop rollouts, TRIAGE also reduces environment-facing turns by an additional $10.4\%$ and $14.8\%$ relative to GRPO.

Problem

• credit assignment
• agentic reinforcement learning
• role-typed rewards
• outcome-only credit
• process rewards

Innovation

• role-typed credit assignment
• process reward
• structured judge
• advantage estimation
• agentic reinforcement learning