RTMC: Step-Level Credit Assignment via Rollout Trees

2026-04-13

AI Summary

This work addresses the lack of fine-grained, efficient, and stable credit assignment mechanisms in multi-step reinforcement learning for agents. It proposes Rollout-Tree Monte Carlo (RTMC), a critic-free method that constructs a rollback tree and leverages trajectory overlap at intermediate states to enable state-matching-based credit assignment at the action level. RTMC introduces a novel state-action signature system to compress interaction histories, facilitating efficient cross-trajectory comparison, and employs Monte Carlo return aggregation for step-wise advantage estimation. Evaluated on SWE-bench Verified, RTMC improves pass@1 by 3.2 percentage points over GRPO.

Technology Category

• Multiagent Systems: Multiagent Learning
• Machine Learning: Reinforcement Learning
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Multi-step agentic reinforcement learning benefits from fine-grained credit assignment, yet existing approaches offer limited options: critic-free methods like GRPO assign a uniform advantage to every action in a trajectory, while learned value networks introduce notable overhead and can be fragile under sparse rewards. We observe that group rollouts targeting the same problem often traverse overlapping intermediate states, implicitly forming a tree whose branches diverge at successive decision points. Building on this insight, we introduce Rollout-Tree Monte Carlo (RTMC) advantage estimation, which aggregates return statistics across rollouts sharing a common state to produce per-step Q-values and advantages--without any learned critic. A state-action signature system compresses raw interaction histories into compact, comparable representations, making cross-rollout state matching tractable. On SWE-bench Verified, RTMC improves pass@1 by 3.2 percentage points over GRPO.

Problem

• credit assignment
• reinforcement learning
• sparse rewards
• multi-step agentic
• advantage estimation

Innovation

• Rollout-Tree Monte Carlo
• credit assignment
• critic-free reinforcement learning
• state-action signature
• multi-step agentic RL