Information Gain-based Rollout Policy Optimization: An Adaptive Tree-Structured Rollout Approach for Multi-Turn LLM Agents

2026-07-07

AI Summary

Existing reinforcement learning methods lack explicit evaluation of the informational value of intermediate states in long-horizon search tasks, often wasting rollout budgets on low-value branches. This work proposes the IGRPO framework, which for the first time employs information gain as a core criterion to adaptively allocate rollout expansion budgets within tree-structured search, prioritizing exploration of high-information branches. Policy optimization is then performed using teacher trajectories induced by this information-aware search process. By unifying adaptive tree search with policy learning, IGRPO achieves budget-aware, efficient exploration. Evaluated on seven search-augmented question answering benchmarks, IGRPO significantly outperforms strong baselines under identical rollout budgets, demonstrating the effectiveness of information-guided policy optimization.

Abstract

Reinforcement learning has become a promising paradigm for improving large language model (LLM) agents on long-horizon search tasks, where the agent must make a sequence of intermediate decisions before receiving a final outcome. However, existing methods still face a key limitation: the rollout budget is often allocated without explicitly assessing the utility of intermediate states. As a result, substantial computation may be spent on low-value states, even though different branches can vary drastically in their informativeness. In this paper, we propose Information Gain-based Rollout Policy Optimization (IGRPO), a policy optimization framework that treats intermediate-state informativeness as the organizing principle of rollout collection. Specifically, IGRPO performs budget-aware tree-structured rollouts by allocating expansion budget according to node-level informativeness, so that more informative branches are expanded more frequently while unpromising branches are progressively suppressed. We further demonstrate that the information gain-based rollout induces an explicit limiting teacher distribution over trajectories, which naturally yields a clear policy optimization target, thereby unifying adaptive tree-structured exploration with principled policy learning under a single framework. Experiments on seven challenging search-augmented QA benchmarks demonstrate that IGRPO consistently outperforms strong baselines under the same rollout budget constraints, validating the effectiveness of leveraging the induced teacher distribution to guide policy optimization for long-horizon search agents.

Problem

• rollout budget allocation
• intermediate state utility
• long-horizon search
• information gain
• LLM agents

Innovation

• Information Gain
• Rollout Policy Optimization
• Tree-Structured Search
• LLM Agents
• Policy Learning