SKILLER: Language-Level Reinforcement Learning for Reusable Skill Extraction in Small Language Models

2026-08-11

AI Summary

This work proposes SKILLER, a framework designed to reduce the inference cost of powerful language model agents in real-world tasks by automatically extracting efficient and reusable skills for smaller language models. SKILLER introduces the first fully natural-language-based reinforcement learning pipeline, wherein a strong language model serves as both policy and critic, while the small-model agent acts as the environment; reward signals are communicated exclusively through natural language, and a novel, transferable behavioral constraint mechanism tailored for small models is incorporated. Experimental results demonstrate that Qwen-4B and Qwen-9B models optimized with SKILLER significantly outperform existing methods across five benchmarks, achieving gains of 1.8–20.4 percentage points, and match the performance of closed-source strong models on single-skill tasks from SkillsBench.

Abstract

Agent skills represent a standardized format for packaging procedural knowledge and domain expertise, serving within agent harness systems as an essential mechanism to continually constrain a language model's behavior space for repeatable, high-quality task execution. However, because strong closed-source models entail high inference costs, current popular agent harnesses, such as Codex and OpenClaw, remain prohibitively expensive when deploying these skills to accomplish real-world tasks. The rapid capability enhancement of open-source models deployable on consumer-grade GPUs presents a compelling opportunity to drastically reduce these costs by leveraging skill-based behavioral constraints. Nevertheless, automatically generating effective skills tailored specifically for such compact models remains a significant practical challenge. To address this, we propose SKILLER, a natural-language-driven reinforcement learning framework designed to automatically generate executor-specific skills for small models, which employs a strong model as the actor and critic, treats the small-model agent system as the environment, and propagates all reinforcement learning signals entirely via natural language. Extensive experimental evaluations across five relevant benchmarks using Qwen3.5-9B and Qwen3.5-4B demonstrate that SKILLER outperforms three open-source and one closed-source skill generation or evolution methods, achieving absolute gains ranging from 4.3 to 20.4 percentage points for the 9B model and 1.8 to 13.3 points for the 4B model, while remarkably matching the performance of strong closed-source models on single-skill tasks in SkillsBench. The project is available at https://github.com/DANG-ai/SKILLER.

Problem

• reusable skill extraction
• small language models
• agent skills
• inference cost
• skill generation

Innovation

• language-level reinforcement learning
• reusable skill extraction
• small language models
• natural-language-driven RL
• executor-specific skills