TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents

2026-07-15

AI Summary

This work addresses the challenge of credit assignment in long-horizon agents performing multi-turn tool use, where sparse, high-variance, and potentially misleading episodic rewards hinder learning. The authors propose a dense credit assignment method that requires neither auxiliary critics nor process supervision. By modeling trajectories as state transitions at tool-call boundaries, the approach constructs state values using the log-probability of gold answers generated by a frozen reference model and computes dense per-step rewards via temporal difference (TD) learning. A key component—single-step log-ratio TD—automatically suppresses redundant tool invocations. Experiments demonstrate substantial improvements: on BrowseComp-Plus, success rates for Qwen3-4B and Qwen3-30B-A3B rise from 7.2% to 35.6% and from 8.4% to 42.6%, respectively. The method also exhibits strong transferability and faster convergence in open-web tasks.

Abstract

Multi-turn agents solve complex tasks through extended sequences of tool interactions before producing a final answer, making credit assignment a fundamental challenge during post-training. Outcome rewards provide reliable supervision for short-horizon reasoning, but become sparse and high-variance as trajectories grow to tens or hundreds of tool calls. They can also be misleading: a failed rollout may contain many useful actions that move the agent closer to the goal, yet outcome-only training assigns them the same negative advantage as the eventual mistake. We propose TRACE (Turn-level Reward Assignment via Credit Estimation), a dense credit-assignment method for agentic reinforcement learning. TRACE represents rollouts as state transitions at tool-call boundaries, obtains gold-answer log-probabilities from a frozen reference model, transforms them into log-ratio state values, and derives per-action rewards as Temporal-Difference changes in those values. This requires no additional critic or process-label training, and its one-step log-ratio TD component telescopes across redundant tool calls. On long-horizon complex search, TRACE substantially improves base-model tool-use ability using pure RL, without a cold-start supervised fine-tuning stage, an agentic mid-training stage, or training on live-web data. On the closed-web BrowseComp-Plus benchmark, it raises Qwen3-4B from $7.2$ to $35.6$ and Qwen3-30B-A3B from $8.4$ to $42.6$. The learned search behavior also transfers to open-web benchmarks, and the learning curves show earlier improvement and faster convergence during RL training.

Problem

• credit assignment
• long-horizon agents
• reinforcement learning
• sparse rewards
• multi-turn reasoning

Innovation

• credit assignment
• reinforcement learning
• long-horizon reasoning
• dense reward
• temporal-difference learning