OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning

2026-06-25

AI Summary

This work addresses the challenges in outcome-based reinforcement learning where trajectory-level sparse rewards provide insufficient guidance for intermediate decisions, and existing skill distillation methods suffer from distribution shift and high maintenance costs due to reliance on external memory or privileged context. To overcome these limitations, the authors propose a fully on-policy hierarchical skill self-distillation framework that extracts both episode-level and step-level skills directly from trajectories generated by the current policy. A critical-first routing mechanism selects key skills to inject into the historical context, and token-level self-distillation advantages are constructed from the log-probability differences of responses under old and new contexts. These advantages are jointly optimized with outcome-based rewards. Experiments on ALFWorld, WebShop, and search-based QA tasks demonstrate significant improvements in agent performance, sample efficiency, and robustness over pure outcome-reward RL and existing skill distillation baselines.

Abstract

Outcome-based reinforcement learning provides a stable optimization backbone for language agents, but its sparse trajectory-level rewards provide little guidance on which intermediate decisions should be reinforced or suppressed. On-policy self-distillation offers dense token-level supervision, yet existing skill-conditioned variants often rely on external skill memories or retrieved privileged context, which are costly to maintain and can be mismatched with the state distribution induced by the current policy in multi-turn interaction. We propose \textbf{OPID} (\textbf{O}n-\textbf{P}olicy Sk\textbf{i}ll \textbf{D}istillation), a framework that extracts skill supervision directly from completed on-policy trajectories. OPID represents trajectory hindsight as hierarchical skills: episode-level skills capture global workflows or failure-avoidance rules, while step-level skills capture local decision knowledge at critical timesteps. A critical-first routing mechanism uses step-level skills when critical decisions are identified and falls back to episode-level skills as default guidance otherwise. The selected skill is injected into the interaction history, allowing the old policy to re-score the same sampled response under both original and skill-augmented contexts. The resulting log-probability shift yields a token-level self-distillation advantage, which is combined with the outcome advantage for policy optimization. OPID thus preserves RL as the primary training objective while introducing dense, distribution-matched hindsight supervision. Experiments on ALFWorld, WebShop and Search-based QA demonstrate that OPID generally improves agent performance, sample efficiency, and robustness over outcome-only RL and existing skill-distillation baselines. Our code is available at https://github.com/jinyangwu/OPID/tree/main.

Problem

• reinforcement learning
• skill distillation
• on-policy learning
• language agents
• sparse rewards

Innovation

• On-policy distillation
• Hierarchical skill extraction
• Token-level supervision
• Distribution-matched hindsight
• Reinforcement learning for language agents