Skill-R1: Agent Skill Evolution via Reinforcement Learning

2026-05-10

AI Summary

Existing approaches to agent skill optimization rely heavily on prompt engineering or alignment with large task-specific models, resulting in high costs, strong model coupling, incompatibility with closed-source models, and a lack of mechanisms for continuous evolution. This work proposes Skill-R1, a framework that leverages recursive reinforcement learning with verifiable reward signals to train a lightweight skill generator that guides a frozen large language model, achieving black-box compatibility across both open- and closed-source models. Its key innovation lies in a two-level Group Relative Policy Optimization (GRPO) objective that integrates intra- and inter-generational advantages, enabling directional skill evolution rather than myopic self-optimization. Experiments demonstrate that Skill-R1 significantly outperforms both skill-free baselines and standard GRPO on multiple benchmarks with verifiable rewards, particularly excelling in complex multi-step tasks.

Abstract

Agentic large language models often rely on skills, reusable natural language procedures that guide planning, action, and tool use. In practice, skills are typically improved through prompt engineering or by aligning the task LLM itself, which is costly, model-specific, and often infeasible for closed-source models. Skill optimization is not a one-step problem but a recurrent process with two coupled levels of credit assignment: a useful skill must improve rollout quality under current conditioning, while a useful revision must turn observed outcomes into a better skill for the next round. We propose Skill-R1, a reinforcement learning framework for instance-level recurrent skill optimization from verifiable rewards. Rather than updating the task LLM, Skill-R1 trains a lightweight skill generator that conditions on the task context, prior rollouts, and their verified outcomes to produce skills that steer a frozen task LLM. This preserves black-box compatibility with both open- and closed-source models while making adaptation substantially cheaper than model-level updates. Skill-R1 proceeds over multiple generations: at each step, the current skill induces rollouts whose verified outcomes are fed back to produce the next revision. To optimize this recurrent process, we introduce a bi-level group-relative policy optimization objective combining intra-generation and inter-generation advantages. The intra-generation term compares rollouts under shared skill conditioning, while the inter-generation term rewards revisions that improve behavior across successive generations. Together, these provide a principled objective for directional skill evolution rather than one-shot self-refinement. Empirically, Skill-R1 achieves consistent gains over no-skill baselines and standard GRPO across benchmarks with verifiable rewards, with particularly strong improvements on complex, multi-step tasks.

Problem

• skill optimization
• reinforcement learning
• agentic LLMs
• credit assignment
• black-box compatibility

Innovation

• skill evolution
• reinforcement learning
• black-box LLM
• bi-level policy optimization
• verifiable rewards