Problem
Research questions and friction points this paper is trying to address.
agentic skill-use
skill evaluation
overlapping skills
process rubrics
LLM agents
SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use
🤖 AI Summary
Current large language model agents struggle to identify procedural errors—such as selecting incorrect skills, skipping critical steps, or miscombining tools—when relying solely on final outcome validation. To address this limitation, this work proposes SkillCoach, a novel framework featuring a self-evolving process-scoring mechanism. It automatically derives multidimensional scoring rules aligned with skill execution from real behavioral trajectories, decoupling process quality from task outcomes across four dimensions: skill selection, adherence, composition, and reflection. This approach not only uncovers failure modes obscured by conventional accuracy metrics but also generates high-quality supervisory signals for process-level learning. Experimental results demonstrate that the evolved scoring criteria enable finer-grained evaluation and significantly outperform purely outcome-driven training methods in enhancing agents’ skill utilization capabilities.
📝 Abstract
Skills are becoming a reusable operational layer for LLM agents, encoding SOPs, domain rules, tool workflows, scripts, and validation routines. In realistic skill repositories, overlapping skills make reliable skill-use difficult. Final verifier success is too coarse for both evaluation and training, since an agent may pass through trial and error while selecting distractor skills, skipping required steps, composing workflows incorrectly or omitting final checks. We introduce SkillCoach, a self-evolving rubric framework for evaluating and enhancing agentic skill-use. SkillCoach derives skill-grounded process rubrics from real rollouts and evaluates trajectories along four dimensions: skill selection, skill following, skill composition, and skill-grounded reflection. It keeps the external verifier as a separate outcome signal, allowing process quality to be distinguished from accidental task success. The evolved rubrics further serve as process supervision for selecting high-quality training trajectories. Experiments show that evolved rubrics substantially improve evaluation quality, expose failures hidden by final accuracy, and provide stronger supervision signals than outcome-only filtering for enhancing agentic skill-use.
Innovation
Methods, ideas, or system contributions that make the work stand out.
self-evolving rubrics
agentic skill-use
process evaluation
skill composition
trajectory supervision
🔎 Similar Papers
No similar papers found.
