A$^2$TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping

2026-05-07

AI Summary

This work addresses the challenge of credit assignment in reinforcement learning–based large language model agents, where sparse trajectory-level rewards inadequately capture the contributions of individual tool calls across multi-turn interactions. Existing approaches often rely on external models or constrain trajectory diversity. To overcome these limitations, the authors propose using intrinsic information gain (IG) as a process-level signal, combined with within-episode normalization to mitigate positional heterogeneity and variance-rescaled discounted accumulation to stabilize advantage estimation. Furthermore, they introduce an adaptive episode-level clipping mechanism based on normalized IG to dynamically modulate policy update magnitudes. This approach effectively alleviates advantage estimation drift with increasing trajectory depth and significantly improves both the accuracy of per-turn contribution assessment and overall training stability in multi-turn settings.

Abstract

Reinforcement learning for agentic large language models (LLMs) typically relies on a sparse, trajectory-level outcome reward, making it difficult to evaluate the contribution of individual tool-calls within multi-turn interactions. Existing approaches to such process credit assignment either depend on separate external process reward models that introduce additional consumption, or tree-based structural rollout that merely redistributes the outcome signal while constraining trajectory diversity. A promising alternative leverages the per-turn change in the policy's predicted probability of the ground-truth, termed Information Gain (IG), as an intrinsic process signal without an external evaluator. However, prior work on leveraging IG signals within the RL training loop faces three systematic challenges: normalizing across turns that face heterogeneous positional contexts can distort the relative standing of individual turns, accumulating a variable number of terms causes advantage magnitudes to drift with trajectory depth, and a fixed clipping range governs policy updates identically for turns with vastly different IG signals. In this paper, we propose A$^2$TGPO (Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping), which retains IG as the intrinsic signal but re-designs how it is normalized, accumulated, and consumed: (i) turn-group normalization: normalizes IG within each (prompt, turn-index) group so that each turn is compared only against peers at the same interaction depth; (ii) variance-rescaled discounted accumulation: divides cumulative normalized IG by square root of accumulated terms to keep advantage magnitudes comparable across turn positions; and (iii) adaptive turn-level clipping: modulates each turn's clipping range based on its normalized IG, widening the update region for informative turns and narrowing it for uninformative ones.

Problem

• credit assignment
• reinforcement learning
• large language models
• process reward
• multi-turn interactions

Innovation

• Information Gain
• Turn-group Normalization
• Adaptive Clipping
• Process Credit Assignment
• Reinforcement Learning for Agentic LLMs