Skill0.5: Joint Skill Internalization and Utilization for Out-of-Distribution Generalization in Agentic Reinforcement Learning

2026-05-27

AI Summary

Existing skill-based reinforcement learning approaches struggle to balance general-purpose and task-specific skills, often leading to excessive context overhead, overfitting, or knowledge interference that undermines out-of-distribution generalization. This work proposes Skill0.5, a novel framework that integrates general skills internally while externalizing task-specific skills. It employs a dynamic difficulty-aware routing mechanism to hierarchically handle tasks: for challenging tasks, it internalizes general skills via privileged distillation to establish a robust cognitive foundation; for simpler tasks, it enforces the use of task-specific skills through diagnostic probes while suppressing shortcut behaviors. This design substantially reduces context burden and achieves state-of-the-art performance on both in-distribution and out-of-distribution evaluations, significantly outperforming existing memory-based and skill-based reinforcement learning baselines on the ALFWorld and WebShop benchmarks.

Abstract

Equipping large language models with explicit skills has emerged as a promising paradigm for enabling autonomous agents to solve complex tasks. Agent skills can be inherently divided into general skills for broad cognitive transfer and task-specific skills for dynamic execution. However, existing skill-based reinforcement learning (RL) methods typically force a rigid choice between full externalization, which incurs prohibitive context overhead, and full internalization, which risks overfitting and knowledge conflicts. To address this dilemma, we propose Skill0.5, a novel agentic RL framework that explicitly differentiates skill treatments by combining general skill internalization with task-specific skill utilization. Driven by a dynamic, difficulty-aware router, Skill0.5 streams tasks into distinct mastery tiers to apply tailored optimization strategies: it internalizes general skills via privileged distillation to build a cognitive foundation for hard tasks, while using diagnostic probing on easy tasks to penalize shortcuts and enforce specific skill utilization. Experiments on ALFWorld and WebShop demonstrate that Skill0.5 outperforms both memory-based and skill-based RL baselines, yielding performance improvements across both in-distribution and out-of-distribution scenarios.

Problem

• out-of-distribution generalization
• skill internalization
• skill utilization
• agentic reinforcement learning
• cognitive transfer

Innovation

• Skill Internalization
• Task-Specific Skill Utilization
• Out-of-Distribution Generalization
• Dynamic Routing
• Privileged Distillation