Learning Agentic Policy from Action Guidance

2026-05-12

AI Summary

This work addresses the challenge that large language models often struggle to obtain meaningful reward signals in reinforcement learning due to insufficient exploration. To overcome this limitation without requiring supervised fine-tuning, the authors propose a novel reinforcement learning paradigm that leverages human everyday interaction data as plan-like reference guidance. By employing a hybrid policy training framework, the method internalizes guided exploration capabilities into an unguided policy and adaptively invokes guidance based on the principle of minimal intervention, effectively balancing task difficulty against off-policy risk. Evaluated on the GAIA and XBench benchmarks, the approach outperforms zero-shot RL by 10.7 and 19 percentage points, respectively, achieving performance comparable to supervised fine-tuning followed by reinforcement learning while circumventing the costly cold-start fine-tuning phase.

Abstract

Agentic reinforcement learning (RL) for Large Language Models (LLMs) critically depends on the exploration capability of the base policy, as training signals emerge only within its in-capability region. For tasks where the base policy cannot reach reward states, additional training or external guidance is needed to recover effective learning signals. Rather than relying on costly iterative supervised fine tuning (SFT), we exploit the abundant action data generated in everyday human interactions. We propose \textsc{ActGuide-RL}, which injects action data as plan-style reference guidance, enabling the agentic policy to overcome reachability barriers to reward states. Guided and unguided rollouts are then jointly optimized via mixed-policy training, internalizing the exploration gains back into the unguided policy. Motivated by a theoretical and empirical analysis of the benefit-risk trade-off, we adopt a minimal intervention principle that invokes guidance only as an adaptive fallback, matching task difficulty while minimizing off-policy risk. On search-agent benchmarks, \textsc{ActGuide-RL} substantially improves over zero RL (+10.7 pp on GAIA and +19 pp on XBench with Qwen3-4B), and performs on par with the SFT+RL pipeline without any cold start. This suggests a new paradigm for agentic RL that reduces the reliance on heavy SFT data by using scalable action guidance instead.

Problem

• agentic reinforcement learning
• exploration capability
• reward states
• action guidance
• learning signals

Innovation

• Agentic Reinforcement Learning
• Action Guidance
• Mixed-Policy Training
• Minimal Intervention
• Exploration Enhancement