T$^2$PO: Uncertainty-Guided Exploration Control for Stable Multi-Turn Agentic Reinforcement Learning

2026-05-03

AI Summary

This work addresses the instability and frequent collapse in multi-turn reinforcement learning caused by inefficient exploration. The authors propose T²PO, a novel framework that introduces uncertainty-aware mechanisms at both token and turn granularities for the first time. By dynamically monitoring exploration states and triggering thinking-based interventions alongside turn-level resampling, T²PO enables fine-grained control over the exploration process, effectively mitigating unproductive interactions and training inefficiencies. Extensive experiments on multi-turn task environments—including WebShop, ALFWorld, and Search QA—demonstrate that the proposed method substantially enhances training stability and task performance, validating its significant improvement in exploration efficiency.

Abstract

Recent progress in multi-turn reinforcement learning (RL) has significantly improved reasoning LLMs' performances on complex interactive tasks. Despite advances in stabilization techniques such as fine-grained credit assignment and trajectory filtering, instability remains pervasive and often leads to training collapse. We argue that this instability stems from inefficient exploration in multi-turn settings, where policies continue to generate low-information actions that neither reduce uncertainty nor advance task progress. To address this issue, we propose Token- and Turn-level Policy Optimization (T$^2$PO), an uncertainty-aware framework that explicitly controls exploration at fine-grained levels. At the token level, T$^2$PO monitors uncertainty dynamics and triggers a thinking intervention once the marginal uncertainty change falls below a threshold. At the turn level, T$^2$PO identifies interactions with negligible exploration progress and dynamically resamples such turns to avoid wasted rollouts. We evaluate T$^2$PO in diverse environments, including WebShop, ALFWorld, and Search QA, demonstrating substantial gains in training stability and performance improvements with better exploration efficiency. Code is available at: https://github.com/WillDreamer/T2PO.

Problem

• multi-turn reinforcement learning
• exploration inefficiency
• training instability
• uncertainty
• low-information actions

Innovation

• uncertainty-guided exploration
• multi-turn reinforcement learning
• token-level control
• turn-level resampling
• training stability