Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents

2026-08-13

AI Summary

This work addresses key challenges in applying multi-turn, multi-step large language model (LLM) agents to reinforcement learning, including sparse trajectory-level rewards, ambiguous credit assignment across episodes, and limitations of conventional distillation methods—such as reliance on teacher policy constraints or susceptibility to gradient collapse. The authors propose CrEST, a novel framework that enables hierarchical credit assignment at both episode and token levels through verifier-constrained reinforcement learning and fine-grained supervision from a privileged self-teacher. Innovatively, CrEST redefines the teacher’s role from dictating update direction to modulating update magnitude, integrating episode-partitioned advantage estimation with an entropy-gated self-teaching mechanism. This approach achieves dense credit assignment while preserving the performance ceiling imposed by the verifier. Experiments demonstrate that CrEST significantly outperforms existing reinforcement learning and distillation baselines on BFCL V3 and WildToolBench, with particularly notable gains on long-horizon trajectories and stringent conversation-level evaluation metrics.

Abstract

Reinforcement learning with verifiable rewards (RLVR) offers a verifier-bounded performance ceiling for training multi-turn tool-use agents, yet its trajectory-level credit assignment conflates heterogeneous per-turn outcomes into a single reward signal. On-policy distillation provides dense per-token supervision but is either teacher-bounded or prone to gradient concentration collapse. We introduce $\textbf{CrEST}$, a hierarchical credit assignment framework that retains RL's verifier-bounded ceiling while incorporating dense token-level signals from a privileged self-teacher. $\textbf{CrEST}$ resolves credit at two levels: turn-segmented verified advantages address inter-turn dilution, while entropy-gated self-teacher modulation refines intra-turn token contributions. Experiments on BFCL V3 and WildToolBench show that $\textbf{CrEST}$ consistently outperforms both RL and distillation baselines across two model scales, with the largest gains on long-trajectory and strict session-level metrics. Our work demonstrates that the teacher's role in policy optimization can be reduced from determining update directions to modulating update magnitudes, unlocking dense credit assignment without sacrificing the verifier-bounded ceiling.

Problem

• credit assignment
• multi-turn agents
• verifier-bounded reinforcement learning
• dense supervision
• large language models

Innovation

• credit assignment
• verifier-bounded RL
• self-distillation
• multi-turn agents
• token-level supervision